{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression\n",
    "   完成 HW4 第五题的逻辑回归。逻辑回归实际上是一个单层线性分类器再经过一个 logistic/softmax 激活函数进行预测，预测结果和真实的 label 来最小化交叉熵损失，并以此计算梯度回传训练分类器的参数。这里我将分别使用 GD 和 SGD 来更新权重，同样的，也使用反向传播技术简化计算。\n",
    "停止条件设置为 acc >= 0.97 ，最后再计算 confusion matrix, precision, recall and F1-score.\n",
    "\n",
    "\n",
    "   由于这个问题的数据集是不平衡的，接下来我们需要重采样后再重复计算以上过程。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 初始化设置\n",
    "导入数据，并查看。手写数字数据集放在 hw4_lr 文件夹内，含有 60000 张training data 和 1960 张 test data. 我们的任务是训练一个 Logistic 分类器识别 '0' 和  '!0'. 可能会需要正则化操作,或者需要调整其他的超参数，所以我们把 60000 张 training_data 又分出 2000 张作为 validation_set, 来进行调参."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(58000, 784)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAACRCAYAAADTnUPWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAFDZJREFUeJzt3Xl0VFWeB/DvL5WwKntIA0EWE2RzoWURtVtbZEZUwHan1cZxQQVFOVEEnWmmT/exdZyxB1xg8Kio06I2qCC0okNjqw2N4AoYCIishlVkC0uWO3+kfLd+RV5SqarUcuv7OadO7q3fq/du8qvcvNy67z4xxoCIiNJfVrIbQERE8cEOnYjIEezQiYgcwQ6diMgR7NCJiBzBDp2IyBFOd+gisklELo5wWyMiBVEeJ+rXUv0xr25iXmPndIeeykSkjYjsFpGPk90Wip2I/KeIrBeRgyKyVkR+new2UezSLa/ZyW5ABnsMQDH4R9UVhwEMB1ACYACAd0VkgzFmaXKbRTFKq7xmTGciIgNFZJmI/CAipSLylIg0CtvsUhHZKCJ7RORxEckKef0tIlIsIvtEZJGIdImhLYMB9AXwQrT7oGqpkldjzBRjzFpjTJUxZjmAjwAMjuFby2jMa3QypkMHUAlgAoB2qE7IEABjw7b5JYD+AH4KYCSAWwBARK4A8BCAKwHkojqps2s6iIj8SkS+8muEiAQAPA3gbgBcdyF2KZHXsG2bovpsbk09vxeymNdoGGOcfQDYBOBin9h9AN4MqRsAl4TUxwJYHCy/A+DWkFgWgDIAXUJeWxBhmyYAmB4s3wzg42T/nNLtkYp5DWvDiwDeBSDJ/lml04N5jf2RMWfoItJDRBaIyA4ROQDgEVT/9Q+1NaS8GUDHYLkLgKnBf/9+APA9AAHQqZ5t6AhgPICHo/ke6ESpkNew9jyO6uG0a02wF6D6Y16jkzEdOoDpANYCKDTGtED1v2QStk3nkPIpAL4LlrcCuMMY0yrk0dTU/4ORgQA6APhaRHYAmApgYPBNG6jvN0QAUiOvAAAR+S2AYQD+yRhzIJp9kId5jUImdegnAzgA4JCI9ARwVw3bPCAirUWkM4B7AbwWfH4GgMki0gcARKSliFwTRRveAdAVwFnBx28AfA7gLGNMZRT7o9TIK0RkMoBfARhqjNkbzT5IYV6jkEkd+v2oTsxBAM/CJj/UPACfAvgCwEIAzwGAMeZNVE8zfDX4799qVP/FPoGI3CAiNX5oYow5ZozZ8eMDwH4A5cEyRSfpeQ16BNVnietF5FDw8VB03xKBeY2KpPBwEBER1UMmnaETETmNHToRkSPYoRMROSKmDl1ELhGRdSKyQUQmxatRlFzMq7uYW7dF/aFocN50CYChALYBWAFglDHma7/XNJLGpgmaR3U8ip+jOIzj5lj4nF4AzGs6qy2vQP1zy7ymjoPYt8cYk1vXdrGstjgQwAZjzEYAEJFXUb2egu8vfhM0xyAZEsMhKR6Wm8W1hZnXNFVHXoF65pZ5TR3/Z+ZsjmS7WIZcOkFfersNNVxaKyJjRGSliKwsx7EYDkcJwry6q87cMq/pLZYOvaZ/7U4YvzHGzDTG9DfG9M9B4xgORwnCvLqrztwyr+ktlg59G/RaCvmwaylQ+mJe3cXcOi6WDn0FgEIR6RZceP56APPj0yxKIubVXcyt46L+UNQYUyEidwNYBCAA4HljTOou/E4RYV7dxdy6L6Z7ihpj/gLgL3FqC6UI5tVdzK3beKUoEZEj2KETETmCHToRkSPYoRMROYIdOhGRI9ihExE5gh06EZEj2KETETmCHToRkSPYoRMROSKmS/+JUt3O8ed6ZXPRPhUbXbDcK49p5Xv/DgDAhO32Rg/bL2+mYpW7d8fSREqgQNs2qi4tW3jlLVd1VLGj7ezKwgW//VLFqsrKGqB1seMZOhGRI9ihExE5gkMuDSjQq1DVixa84ZUfu+lGFZOl+l86ilx2vr2LWtkL+i29oveTXrm4vFzFJm68yiu/t6uXis0seE3VZ+R/ZMt/66Ji83u3rWeLqSFl9e3plddPbqpit5y+VNWL2i6KaJ+98u5U9cKbP42ydQ2LZ+hERI5gh05E5Ah26EREjsjYMXQ5u4+qZ5Ud98qVxevjc5BnDqvqD5XNvXL2D0dUrDI+R8xI/RZs8cpXt1ypYj3m3eOVe/9ui4qZ0u2++xwz4C5Vnz5nule+reVGFfvjf13mlU8t+kcELaZYyYDTvfKGCQEV++D8p7xybqCximWFncMuLGvtlTcea69i41qv88ov//xZFfvdgNFe2axYFWmzGxzP0ImIHMEOnYjIERk15PL9LYO98utTHlexYbMmeuUuU6Ifcjk2bIBXnlc4TcV+OmeCVy74mv+aR+vQteeo+pTcp73yOZ/dpGI9xn7ilSvqcYzwf6OHvHW/V15/9TMq9tjwV7zyzKLu9TgK1SaQm+uVS6Z2UrG3z7U56J6TE/bKxvDzwoHOqv7WVed75arGej/jFtghl/6N9aDokTw7HbKJ79ESj2foRESOYIdOROQIduhERI7IqDH0Rtfu9MojPhujYl2mLA3fPCqbR4hXbiaNVKz7nKNxOUamqwwbMn3pgB1fDcxtmMvwT/1zSO6u1rHc7AP2+O308Sv37G2Q9mSC7TfapTPWXDA1LBo+bl6z/w0fM7/iXFWvXFfilaWfnsqcjniGTkTkiDo7dBF5XkR2icjqkOfaiMj7IrI++LV1bfug1MO8uou5zVyRDLnMAvAUgJdCnpsEYLEx5lERmRSsPxj/5sXXH3rY1Q5Hvzemli2jd9JPDnnlLEgtWybdLKRpXlu/pacUzn27h40dWNYgxwwc9Z/0eF7jKq+8+fbTVCz/D/EZyqunWUjT3IbqNGJTRNvNOfQTVX+ixN6MJG+iUbHKdf5Tkved3sI3li7qPEM3xnwI4Puwp0cCeDFYfhHAFXFuFzUw5tVdzG3minYMPc8YUwoAwa/t/TYUkTEislJEVpbjWJSHowRhXt0VUW6Z1/TW4B+KGmNmGmP6G2P659RyBRelF+bVTcxreot22uJOEelgjCkVkQ4AdsWzUfESKOim6rlZDT+e+d7ZM71yadgSioEj9o45emQvZaRFXqsOH657o3hbZcden/xBX95/Tyu7+mJZd31XpBSSFrlVbrd/UHqPu0eFOr9vf7mar9mhYu0226mI9VnFtCwvpT/ziki0Z+jzAfy4fuRoAPPi0xxKMubVXcxtBohk2uJsAMsAnCYi20TkVgCPAhgqIusBDA3WKY0wr+5ibjNXnUMuxphRPqEhPs+njMoN36r6quMdvLI009PQspo188pVZWVRH7N9wO7n93v66uCXdvW20OPFesxopHNek8Ecsx8QHqpMpfX1TuRKbkN/fwsmfOu7XX1W0axN+YCDcdpT8vBKUSIiR7BDJyJyBDt0IiJHZNRqiw/+9TqvXDJ8uoqd9+frvXLbyfqms1VfrY3qeKNarVD1bz4aZst/7K1iJ73OOxilstDPPNpl7/bf7lDAN0aJseU3dkXFimZhE4TDZyaGhK8s9F824u5tF6p603c/q2kXScczdCIiR7BDJyJyREYNufT+981eeUg3fZeCZWe95pUD7+q/cwM/v8Yrlx3TN604L3+jqgfEvvbU7KYq9sl7dhpjl9eTsgofRcn0tleH3t7yY9/tTlkU+bWJ2fn2xhz7z8lXsR2D7Puo4DU9nc6sXI1ME2ihV0I8OtDe/CJn8k4V+6rnk777yRE9JFZu/PO15IgdZts25hQVMxXF/o1NIp6hExE5gh06EZEj2KETETkio8bQK3bYsbZml+lvvf8dd3vl5sP16m2X5a/x3WffpttUvdLYu9ecO3mcinWdvdIrp9JUJzpxKQYUdlHV7RdEdjebof/xoaq/dMtAr3xjTz2N9YymS7zyZc0OqdimCrsUxIjud6hY/lURNSXtSGO9XO/xC073yhOeeVnFftF0sVfeWanXbV9yxN5d7zclI1Vsdp9Zqt4x23+J4CZZduXMjde2UrHu6+zyD1VHU+fm7zxDJyJyBDt0IiJHsEMnInJERo2hhzIVetHN9k+HzAt/Wm+7FHrueag3x9+o6lc8+JRXbvfhdhWrKD9ez1ZSTbJOPlnVpbNdFnnX4LYqtneAnWc8apD/8grtG32n6ve08p9rXpv72qxS9dPOKvXddsLbv/bKUxfqOx012mnvypS/2v8znHSX1cSORe+9rp+KffTINN/X9Zlt72CUv0TPJW+80H5W0baD/mxi9qKzVb2orf+c/kGNbU6+ulm3ZfDW8V4576UvVSzRS2GH4hk6EZEj2KETETkiY4dc4uVg16q6N6J6Cx9WWft4L698/8/fUbE7W/4tqmN8U3HEK28q19PSjhg9PNZU/Ifden34L175lJn60vLAks/CN/cUwH8IyNV3VfjUxLVPnGHLI/2HWEauu0LVezxul9yo3Knvd53d2S6jcOb8LSr2QNuvVX1/lc3zoLlFKtahp93v4tNfU7Fl/2bbet2oy1VszzQ73bLJXv+bhgc+8H9vRItn6EREjmCHTkTkCHboRESO4Bh6PYVfIj7qor+r+hP77LKeVXv3JaRNLmq6sImqbzh1hlfeV3VExS5da+9EtX5rnop1XGDf4oGjesGF5iV7vXJlyTcqtqlYj8ve2sIu8fDqoVwVKxhrx2kr9zHn4STb5mDdf5+pYmtH2DnC2yr0Jfwj/meiV+76vM5PRci4efnFeipi38c+98pT2n+qYi8c0Es6vPzwcK9c8Ib+TCPQzk6BvXDoPSp2+Lr9XvnNfs+qWP40/+UEFhy2+5zZo7vvdtHiGToRkSPYoRMROYJDLvWUlaf/3Z6S+5Gqn/bWWK9ceHB5QtrkojcK3lf11w/ZFfRmjrlNxUKnfxVCr35Zm6ocOxWxZMZAFbu0+ROq/o9jdqjthTv1Cn6BffGffuaSrQ/Yn+3aEVNV7LuQYZZrHn1Axbq+Zacmfn9RNxUzN9pprXP66n3mBuyQR59X9VBJj5l7VL3ZOv/f0co9dkiuxey9KtZiti1fPXaiiuVdvRm+ikKnx8b/CmCeoRMROaLODl1EOovIEhEpFpE1InJv8Pk2IvK+iKwPfm1d174odTCvbmJeM1skZ+gVAIqMMb0AnANgnIj0BjAJwGJjTCGAxcE6pQ/m1U3MawarcwzdGFMKoDRYPigixQA6ARgJ4MLgZi8C+ADAgw3SyhSy5apOtca7za2oNZ4qUj2voXd+AoDiI/bnnv13vUJepHd/ymreXNWr5tuT1A09Z6jYvipR9UlFd3nlZktS97ORVMzr9Nuf8Y01CfkxD79T3+2p03g7BXR0i7drOYKeJtjnFbsSYsFkfZeoyor4/362f2apqhv/bxfA9tqCMavXGLqIdAXQD8ByAHnBN8+Pb6L2Pq8ZIyIrRWRlOY7VtAklGfPqJuY180TcoYvISQDmArjPGHMg0tcZY2YaY/obY/rnwH/CPSUH8+om5jUzRTRtUURyUP3m+JMx5o3g0ztFpIMxplREOgDY5b8Hd5SdeaTWePZfP601nkpSOa/PHchX9X9tZ4dZ+r4yWsU6trZX7X27pqOKnbzJnrPcdttCFRvT6gOvXLRjsIqtLjpD1Zt9kLrDLOFSLa8fHurplQc11jcAaRMyxfChdl/47uPytVeq+pZl9v3Rfc5+FStYY38Hw29k47pIZrkIgOcAFBtjQifnzgfw42/WaADz4t88aijMq5uY18wWyRn6eQBuArBKRH78E/oQgEcBvC4itwLYAuCahmkiNRDm1U3MawaLZJbLxwDEJzwkvs2hRGFe3cS8ZjZe+h+Bqp/Zm9eWXPScio3dfn7Y1rWPsVNk5vbSkzAenWZXxfvkl/qy/BwJGTnsCV+XrNI39H7l98O8covZYSvtgZfzx8vSX9jPNQbdcJGK7T/T3jEoe3eOivWYYaf4Ze/QQ/5dj271yq7e3SkavPSfiMgR7NCJiBzBIZcIHG1nV+WrCrsu8b1VfVS9B1YmpE2ZpnC8nTZ4w/jzotpHC3wT9kx4nRpC5d7vvXLeNH1VZV74xiEya8JhfPAMnYjIEezQiYgcwQ6diMgRHEOPwLZ/tuPmnx/Xk6R6Tdyo6pUJaRER0Yl4hk5E5Ah26EREjuCQSz3N2vMzVQ+dkkVElEw8QycicgQ7dCIiR7BDJyJyBMfQI9Djzk+8Mi8WJ6JUxTN0IiJHsEMnInIEO3QiIkewQycicgQ7dCIiR7BDJyJyhBhj6t4qXgcT2Q1gM4B2APYk7MC1y8S2dDHG5MZrZ8xrnZjX+MnUtkSU24R26N5BRVYaY/on/MA1YFviJ5Xaz7bETyq1n22pHYdciIgcwQ6diMgRyerQZybpuDVhW+InldrPtsRPKrWfbalFUsbQiYgo/jjkQkTkCHboRESOSGiHLiKXiMg6EdkgIpMSeezg8Z8XkV0isjrkuTYi8r6IrA9+bZ2AdnQWkSUiUiwia0Tk3mS1JR6YV9UWZ3LLvKq2pEVeE9ahi0gAwNMAhgHoDWCUiPRO1PGDZgG4JOy5SQAWG2MKASwO1htaBYAiY0wvAOcAGBf8WSSjLTFhXk/gRG6Z1xOkR16NMQl5ABgMYFFIfTKAyYk6fshxuwJYHVJfB6BDsNwBwLoktGkegKGp0BbmlbllXtM3r4kccukEYGtIfVvwuWTLM8aUAkDwa/tEHlxEugLoB2B5stsSJebVR5rnlnn1kcp5TWSHLjU8l9FzJkXkJABzAdxnjDmQ7PZEiXmtgQO5ZV5rkOp5TWSHvg1A55B6PoDvEnh8PztFpAMABL/uSsRBRSQH1W+MPxlj3khmW2LEvIZxJLfMa5h0yGsiO/QVAApFpJuINAJwPYD5CTy+n/kARgfLo1E9NtagREQAPAeg2BjzRDLbEgfMawiHcsu8hkibvCb4g4RLAZQA+AbAw0n4IGM2gFIA5ag+A7kVQFtUfzq9Pvi1TQLacT6q/339CsAXwcelyWgL88rcMq/u5JWX/hMROYJXihIROYIdOhGRI9ihExE5gh06EZEj2KETETmCHToRkSPYoRMROeL/Acsn/QsXkunNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 导入并观察数据\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np \n",
    "import scipy.io as scio\n",
    "\n",
    "# 从 训练集 中分出一部分作为 val set\n",
    "train_data = scio.loadmat('hw4_lr/train_imgs.mat')['train_imgs'][0:58000]\n",
    "train_labels = scio.loadmat('hw4_lr/train_labels.mat')['train_labels'][0:58000]\n",
    "val_data = scio.loadmat('hw4_lr/train_imgs.mat')['train_imgs'][58000:60000]\n",
    "val_labels = scio.loadmat('hw4_lr/train_labels.mat')['train_labels'][58000:60000]\n",
    "test_data = scio.loadmat('hw4_lr/test_imgs.mat')['test_imgs']\n",
    "test_labels = scio.loadmat('hw4_lr/test_labels.mat')['test_labels']\n",
    "\n",
    "print(np.shape(train_data))\n",
    "# 数据已经预处理称为了一个 28 * 28 的行向量，我们将它恢复成 28 x 28 并观察。\n",
    "plt.figure()\n",
    "plt.subplot(1,3,1)\n",
    "plt.imshow(train_data[26].reshape((28,28)))\n",
    "plt.title('label: %s' % train_labels[0][26])\n",
    "plt.subplot(1,3,2)\n",
    "plt.imshow(train_data[16].reshape((28,28)))\n",
    "plt.title('label: %s' % train_labels[0][16])\n",
    "plt.subplot(1,3,3)\n",
    "plt.imshow(train_data[5].reshape((28,28)))\n",
    "plt.title('label: %s' % train_labels[0][5])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAACRCAYAAADTnUPWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAHNVJREFUeJztnXuMXVd1xr9179x5ejx+xE7sxInzcGICVElx0qRUAhUShdAqtCotULWpiJRSoCWIViQgtapUoVRUqEigVqGhCRLi0QJKRAmPWlRtVB4xJS9iJ46dlxM/4ldsj+d57+4fc5m91rpz9tx75859nPl+0mjOmX3O2fucdc6es7+z9loSQgAhhJDep9DpBhBCCGkN7NAJISQnsEMnhJCcwA6dEEJyAjt0QgjJCezQCSEkJ+S6QxeR50Xk7XVuG0TksibraXpf0ji0az6hXZdOrjv0bkZE1onIqyLycKfbQpaOiPyDiOwVkdMiskdE/rjTbSJLp9fs2tfpBqxg/h7AbvCfal4YB/DbAJ4BcA2A74rIsyGE/+1ss8gS6Sm7rpjORESuFZEfichJETkoIp8TkX632c0isl9EjorIp0WkoPZ/v4jsFpETIvI9EbloCW25HsAbAPxrs8cgc3SLXUMIfxNC2BNCqIQQfgLgfwBcv4RTW9HQrs2xYjp0AGUAHwVwDuYM8jYAH3Tb/A6AHQB+FcAtAN4PACLyLgCfAPC7ADZgzqhfWagSEXmfiDye1QgRKQL4PIAPA2DchaXTFXZ12w5h7m3uFw2eC4nQrs0QQsjtD4DnAbw9o+wOAN9S6wHATWr9gwB2VpcfAnCbKisAOAvgIrXvZXW26aMA/qm6/CcAHu70deq1n260q2vD/QC+C0A6fa166Yd2XfrPinlDF5HLReTbInJIRE4B+BTm/vtrXlLLLwDYXF2+CMBnq8O/kwCOAxAA5zfYhs0A/gLAJ5s5B1JLN9jVtefTmJPTfj9UewHSOLRrc6yYDh3APwHYA2BbCGE15oZk4rbZopYvBPBKdfklAH8aQlijfoZC4x9GrgWwCcBTInIIwGcBXFu9aYuNnhAB0B12BQCIyN8CeAeAG0MIp5o5BpmHdm2CldShjwI4BeCMiGwH8GcLbPNXIrJWRLYA+AiAr1X//s8A7hKR1wOAiIyJyLubaMNDALYCuKr689cAfg7gqhBCuYnjke6wK0TkLgDvA3BDCOFYM8cgBtq1CVZSh/6XmDPMaQBfQDS+5gEAPwPwKID/AHAvAIQQvoU5N8OvVod/T2LuP3YNIvKHIrLgR5MQwlQI4dAvfwC8BmCmukyao+N2rfIpzL0l7hWRM9WfTzR3SgS0a1NIF8tBhBBCGmAlvaETQkiuYYdOCCE5gR06IYTkhCV16CJyk4g8LSLPisidrWoU6Sy0a36hbfNN0x9Fq37TzwC4AcABAI8AeG8I4amsfUoDI2FgZF1T9ZHWMTV+HDNT496nF0Bzdu0vDoWh0tiytJXUz8TMa5guTyxoV6Bx2/b3DYeh/jXL0lbSGKcmDh4NIWxYbLulRFu8FsCzIYT9ACAiX8VcPIXMB39gZB3eeOMdS6iStIInvv+PqeKG7TpUGsP1F3Z1VFEAgFSa9+gKhcx+smv40YtfWmyThmw71L8G111+W0vbSJrj+4/93Qv1bLcUyeV82Km3B7DA1FoRuV1EdonIrpmp8SVUR9pEw3adLk+0rXFkSSxqW2PXWT6vvcZSOvSFXllqXoFCCPeEEHaEEHaUBkaWUB1pEw3btb841IZmkRawqG2NXfv4vPYaS5FcDsDGUrgAMZZCd7AMk6bEHTIsx0hcOjq8b5tdlyKBtJt2t3WZJJ7uf2abodKi4+TA528pp/AIgG0icnE18Px7ADzYmmaRDkK75hfaNuc0/YYeQpgVkQ8D+B6AIoAvhhC6N/A7qQvaNb/QtvlnSTlFQwjfAfCdFrWl2UZkFnl5RKuFNWUZ281tGzLLaupUw7/gxz9qFB28rGLKfAN0Wary1gzTW2nXtkgV9UprnYhblLJJoix13ZYix3TFM5tFQjqRlO0asWtq20oD11XZruZZ1rRZxsmBakQIIQRgh04IIbmBHTohhOSEJWnoHUPpYI3o5FqXFJ8fKCy83WJlKS3e691G+yyEzLLgktHZ/RJ1eH2ws+6PraVZDTWpmYb6tlsMfZ2b1bcbsJW+B3thBquhEZ08JOyT+jaT2q8RtE28fdR1l5rvYepZTunyy/A6zTd0QgjJCezQCSEkJ3Sv5JJyR1TDtqSsMuvKyrGsUHbSyawqm7HjQilX1HauLDXyK4pbj/8/KyX7v7TSJ5llWoIJzmJGqvFyjNGK2jc0b9pVsV7pJDX8Tg3bK9Z2Qa830mYnc0hqaF4sZpcVEvuljqk364XZtuZ5bUBWKVcyy/Qz6e2a2i95j9XYTj1QBf9MZpfp/RqSY1rwes03dEIIyQns0AkhJCewQyeEkJzQvRq6QirZ64VZr62FzDKtjRem7UEL09GPUaZm7DGnohgv07asRr/TGlmf8z8c6I+7DdpLXxnoU8t2v3J//L8bnC9kxRwmW5NblqiQGWhdP6nxNuJuWK8uWrb+qOVN58Tl1f22rJT9TcN/Gymdjnbv2/OirV6fY9G9I6n7Q2q0VmVnv1+KbndH9Y9E4juG+XbhymS2nFkGVRZm3ccyve7uB3PNAUipFDfdstGUzYzG+2XwmUOubcoG7jkP+qNXSl+3JVZTb/JVm2/ohBCSE9ihE0JITugeycW7JdU949PJKtr90MsqU3H4VZi0w7TC5HQ85tlJ2zS1Xpm0ZTVDOjUclvPPM0W7/zwmUr743+1+/cfccTMou//B1mUue/ZpOyMNtsRtsVxJlLlr3hdv44nXbTJFsyNx+Ns37q75ySl1TNvm8iorz1SU7DV7xRZTVnzs2fllKWU/Uv6qGAnGn29KgulGySXlmqjlEn+eWpLyZVpymbFSp5FZvAw6MBAPcbm11cyYtev0qJIznZtxuaRdie19NfzYS8jCTN5O9bDexi2wK9/QCSEkJ7BDJ4SQnMAOnRBCckL3aOieRIRDrZPXaOhKN9eaOQAUJ6LWNrNm0JSVtM536Iwpq5w9G5s1PW3KUhEOD/7Wubb+M7GO/mPjtt1Kw0ciZIAU3bcGte5dGo1ou8yya926eWqqd3IauHJTcy5sZ39F6Ztut5F9p+LKc1b3rEyo7xbBHrNveNisz169Le7Xb93U+tS3kvCyc28T9c4k7v1Jfwvoa+BR1NemG/X0BkIzGN181n0bUbp5cBq6DA3Fzdw3jbPnDaBeSuPaHdaWTY9FO0+us/YZWhXvD5mYMmUmBkebs2TxDZ0QQnICO3RCCMkJnZVcEokqTERFn4wiNVNURUMszNgdj78xug0evdG6CW78Tixb86ydCahdA2XADef8TLBtF80vnrnU1r/hx3Hb4qFjpiyMjsQVHwmykn2d6qYbh+aeeqMmrl9jd1PuZcMvT9jd9j638PEBiIma6Gb7TdlhdOlwlG6mL7D1z24YjUc5cNC1u7LwMgA7lTchUxS70Hap2aANJapIRE1UszqnrrzAFE2eU0ImCZmjdNbWMfzw0wvWBwDTN14RD+l7Su2eOm7vuY4kI6/CN3RCCMkJ7NAJISQnsEMnhJCc0D1uiw3ITkZHTmYssnrZidfH5aEnh0zZ2u/viSuDXieP2pr4CIpOLzt0XdRXw4B1tVr3xGk0g/ZGrImaKBnLC613gmYTONdsmq3D9p2J3yqKR06aMvMVoybTkLr9/bcQz8lou3DhWts2lW1KGnE/tI2x66nkz73wPSSDGvdWve4jZV6wYX55fLPVzGu+qyl0yIu+SVvf8E+ft9sqN2QZHbVlPfi624NNJoQQshCLdugi8kUROSIiT6q/rRORH4jI3urvtaljkO6Dds0vtO3KpZ7x4X0APgfgS+pvdwLYGUK4W0TurK5/vPXNW5zapLNq2blBzW6OrmgjL9qZojrQfc2sPe2i5Ie7rv7JdXr47aIfavczX4eOvObd1HSdbigedEKN1DC9lvvQabvWSAd1SjBHrawyePDI/HLFRd6TZJLm7GS+ISEHedlLJ8oonXuOLXxVuaf6maL10rjEch86bdsm8Nd8YtNwxpZWDim6YIul16IeM/L4K3a/GTfTWz/3a6zkoiXLmmfLPJMNRE1MJQZvAYveYSGE/wZw3P35FgD3V5fvB/CuFreLLDO0a36hbVcuzWro54YQDgJA9ffGrA1F5HYR2SUiu2amxrM2I91BU3adLp/N2ox0D3XZ1th1ls9rr7HsH0VDCPeEEHaEEHaUBkYW34H0BNqu/cXsoTHpLYxd+/i89hrNui0eFpFNIYSDIrIJwJFF91iMhMuddx9KJTzWmvrsmHVNlNS8ea1p91sXKZPs2UeLcxHiJrbHkAJhyiWk1W6UPluJ0uFq9LpUsueES2MTiaGXwa7pbw5N4ZICB5/pJqN+cVO7jZ65SLTIoCJu9k1Ym88OxeOGYevyKindvL3uh623bTOkwgI4G6zaFUNwDF5kBxR9h1+LK5M2TIN2RVz0btPf2fqzwwnU9EGJ7y/Grgkbh5pvOpmb1k2zh3gQwK3V5VsBPLD0ppAugHbNL7TtCqAet8WvAPgRgCtE5ICI3AbgbgA3iMheADdU10kPQbvmF9p25bKo5BJCeG9G0dta3Ja6Sc4UVZEKS6fsR53KeJzFWbG5Ys1wq0by0IlsfRB+P9ybif8jBw7ZyysqwmIYc7PSlKticHJMzdCsBXSjXQ0+8l6ziaeTdejkINkJNQAgqHWfzFhCyjUy2+V0uei4bRu5X80MYHtdw3h8tgpP7LNlWj7zUprG29HLdTNxvbzGSrRmO//qW1xe98Nm4UxRQgjJCezQCSEkJ7BDJ4SQnNDZaIt62nqNGL7gYu0hGsiOMvZUPN3T19osI/tXb55fvvC7Nipi38tK+3aZbMIZq9PL5Nb55ZlVtv7nPnDZ/PKGx6yWN/K8TUydRY3nZeeSo3QOn/lH6+0pN7GUTu40e6+11t+2FWIQ78ZXUe6hjWQsqru+xLun08kr21TSaJ9ZyW1bUN/cpkdcd6iaOvCaPVDhyIm40myEzWWAb+iEEJIT2KETQkhO6J6xQiOY2WWpMju82/yfR+eXn1tjo+LNbo8zAfdtt25QI49snV8unbHHnF7tQ+/FBoV+u+2aZ2LZqidftc0esdEfNTVD2IwycVNDe27wn0w0nJgNqofj3t1R413YlMtacGWouKH5cAxvUB5wCaW1Z+JxK9clbaASqcxsttFsD10X6yvafObY/JCNINjTpOyl8IlDRLn9nt1+rimbGov2qUkin8q34l5vi9Nx47Fd7prXKx0lkn9LwUVjbUFGGr6hE0JITmCHTgghOYEdOiGE5ISe0NDFux6Zqf8JLcsnBZ6ILoeXfOmAKXttx6b55RPbrEY6vTouzw7aY5a99K0ad+nXbHaU/v2HY7OH69fMzVrNFPjumXbc9Xi9NqHLFwatfQprVfLvmmiYcX1ym9VzCzOxjskNNt7ExLr4PnX6YnvIsaviN5afXv1vpuydD3V5bopm3Rb9FP6N8cE7ucNe13J/vOZa6waAvol4zQePWDfjyY02GmZFJfiuiU6q7DpzwXpTVNp/KK6kzrcR9O3Y5Ks239AJISQnsEMnhJCcwA6dEEJyQk9o6KkQuSm/Uj9dWGce8iFQxx6JfqZjD1vtO0yqLEQuXO70m19v1vf/Xqyj/4DP05sg4T9vaCAEq742TWQvWn6a1R69T7LyC5cRmw6vMhpDopZHrIZtMkg5ZoZd9ppEBnh9bcvDPpRrXNd6LQCMvhTT1Z/zmL2vKt+O5/HOQ7dktqXnUbp5ZbPVqY9eHTV0/x1t7Z44b6T0yglTVn456tviQlEX3/pGu+2AzhRm69A6/fErbWjddZWo6ZdeaeA5X2b4hk4IITmBHTohhOSErpVctFyQmvpeu6PO/OPcFkvxdIO4qd5qGO2Tvpo1l/R3fJMbmpfUdPI+nyTa1WkKm8tsY7IZdXCuv5YhpBXZhZwL28xl0a10ZrW95hU1NPZm1W2pMfl0HMdr98K5je2qPj8vXw0ciZE7i6+etPtNqKieXirSkqA736KWClKRBrsESUiG5n5w53L66mjXU1vs9SnMxP1GDlv79O07OL8cJmxshMLqVfPLZ3/tUlM2PZqd3Wh0v42cOr0uuq6evsC27cg1I7EtE1bmG9sfJdtKyZ7v2Q3xOOv/r/VSTfffKYQQQuqCHTohhOQEduiEEJITukZDT2biSZYl9FqfvUavFxP/y1xYS619itM6T2y3dZSORn1XhxqoqT/RtuDKUvptina6Kjalmyf2OXXtFrOuQy74qd6Dx6L7X9/4jCkrHj2lDuLC556Nrm9h/KwpK79pu10fjHYvTrnsNfteisfx3z9SGek1XZQ5vi7qi3oLwF6TE9ecZ8rObInPYZ+VsLH+yfj9of+5I/aYOqPURuvueOYNG+aXZ4btc14atw0ffTRq8ZXDNqT1kHKBHTyw0ZSd3j42v3z8CmvjY2+K69e8YZ8p+/olO+eXb77hD9Bq+IZOCCE5gR06IYTkhK6RXDzGDcrPFE3JLKnIbjrannch1Nv67DV6eDdko/BNn2O33fDjONzSM0wBQIbUbDMn+RgXy5qZiEpy8UN6yVjuVup0QZ1cY6/P0Il4nUcfPWTKwsnX4oqTcaw6l30/yMVW4vHuZtrFsbTHRurU2Y6k4B4pyXaHTboj9poEk3juTlwbZ1WevNyec7/y8tz4M5sove+lmGGssna1KZu48Py4vN65g6p7YM0em0GqsM/arjweZR0/qxQqo5UcsnLM2LHY8NIZe+8cHIjtefTA+abs+s9/YH55LewM11a8XvMNnRBCcsKiHbqIbBGRH4rIbhH5hYh8pPr3dSLyAxHZW/29drFjke6Bds0ntOvKpp439FkAHwshvA7AdQA+JCJXArgTwM4QwjYAO6vrpHegXfMJ7bqCWVRDDyEcBHCwunxaRHYDOB/ALQDeWt3sfgD/BeDjy9JKPw07pS8qmbwmmp7WxNQyAKOph2nr+oap6H44cdWFtmll25b1P1eioHdZ06EHSi5zvFoPLiqfWXf/gnWEuIZcGlts15ZP/XfoaeDhtNVatW5eo5P77yGKcEVMEzQzZjPZ+O80pWdeVvVbXRYlFYrA21yvp9xoGwj3kKIrnlfH4V+P13LgqCtUl/nUJXYKvWyNz9rsgL0+BRVxVS8DwJqnoqtq5fGnTVnZZakybsj++4e2ScKuw09bl8pLns0O6YCi+q62DN9JGtLQRWQrgKsB/ATAudWb55c30caMfW4XkV0ismtmanyhTUiHWapdp8sTC21COsyS7TrL57XXqLtDF5FVAL4B4I4QwqnFtv8lIYR7Qgg7Qgg7SgMji+9A2kor7NpfHFp8B9JWWmLXPj6vvUZdbosiUsLczfHlEMI3q38+LCKbQggHRWQTgCPZR2gcI6v4aaTGVc+7gmUf0yS1cLKKTlwRpuwMzzAdo6eNn2svmY/aKAdiImhZZR+IMBgTLIQBFzGwpBMh2JOoFLPdFmvcGBuglXatW2ZJyQzKPMUZe7zpVSpxyOXWTUzLI8VTzlX0bFyfPW+NKauoxBh9p63NC/tfsduqWaXioybq9YKLsGnO192cdc4cbpROPK+uAW49LpYHnFupkhOnR+1+RWWSkYP2eR08FEcP8qJ1Yy2fVLKnl+B825Qk4u2KUn92mYqkGry7o7ZzjZtx9ozwVlCPl4sAuBfA7hDCZ1TRgwBurS7fCuCBlreOLBu0az6hXVc29byhvxnAHwF4QkQerf7tEwDuBvB1EbkNwIsA3r08TSTLBO2aT2jXFUw9Xi4PI3sO4tta2xzSLmjXfEK7rmy6dup/KimvnpYtzmVJykrbmnEuazpJtNczC9l6ZmFbdG979Xp7zIEjTjMdjdlSwrANE1AZjppcecDuV+mP7Sn3Z4cFCD54X2rqf7dPH09MhV+/83lTNH15zGwzscle1+Kkyjy0yn6bKJSjPUrHbETF4nMxY0zluMs05HRR6U/pqToap7uvjFtc4p5L0e12BJLfRq74QnTzPH3pqCmrlOK2g0etTj60W2Ulcq6qFfWdK3jXVJVVTHzWMv/9Q9kVA851tV/dSyXv0pjIKKXvgTbbjlP/CSEkJ7BDJ4SQnNA1kkvNLEe1XnEzJ00IvQH/Pyn7lAoJNzHRcowehgGY3BQjvZXGrHtb8UXrmlhZHWe7VQada+JQbFvZRfPTMlLNTFE1aq91W1TLHRyZZ7lPNjRrNDEzr/+ZOPwuVdwMYB0N0w2/g54RHLIjMZrhNWoTmZght3dN1MnA/X4pN80WuC0uxW11SfgZy5VEkvWJ6Pa7+nEbtdAkTnezt00SCyd56Oq95GJUyJQdASuXlVzCd+2a6BK+G5nFy2wJuxpXxWV4neYbOiGE5AR26IQQkhPYoRNCSE7oGg29RmvyiZoV2tVJuyjN7add/Jz7oZpeL8NWJy9oF0fnCvnCO+K2b7nYRm97/l+uMOuzYzGuic96o9vt25ZyTTTn5LXLVBiEbiTVxjrLxGvoSs8MrkzU9G2ERGZjSbsUGl04pZOn9NSaOhMaeq+hv+W490Rz7SoJTdm7imrduuxsp7NN+fvBtCsxLd/X6cpCoizl5pyc3r/Mr9B8QyeEkJzADp0QQnJC90guHpMYOSG/OBc/M3J2wyQ9A9MHxUdFuS+5oquu2zu/fO+FD5uy3yxuN+uzIyqJRU0yiuZkFTNrtsdH5gY/HNVuhTWRCXWZj9KoZgbWm0B8sbY0Ul7n8LvhOnuVlEtjKuKkSxYT9Lbedmo95Rpbm1S9AXukZnwmjrPcrokp+IZOCCE5gR06IYTkBHbohBCSE7pXQ09QozFrlH5V8TKXktpCOVsD8zrs0btjtMW3yO12YxugzbbNS8TGTc2XpfbTZb2lu3oNMxkKQJ/bYplmNMXEfq2iFVp4i2zXsen+LcC78Zk17/KZsqVODO4jkKZIXbtmdfKaOhpoT4vhGzohhOQEduiEEJITekNySQ19CtkR9FBMySqJ+lrlG5g4TLKKHpNVGkHLBXXLL42UJStvkRzTBvv0sqxSQ+K1MaQytCQmgKIRmaUV9Mirb480kxBCyGKwQyeEkJzADp0QQnKChOVy81qoMpFXAbwA4BwAR9tWcZqV2JaLQggbWnUw2nVRaNfWsVLbUpdt29qhz1cqsiuEsKPtFS8A29I6uqn9bEvr6Kb2sy1pKLkQQkhOYIdOCCE5oVMd+j0dqnch2JbW0U3tZ1taRze1n21J0BENnRBCSOuh5EIIITmBHTohhOSEtnboInKTiDwtIs+KyJ3trLta/xdF5IiIPKn+tk5EfiAie6u/17ahHVtE5IcisltEfiEiH+lUW1oB7Wrakhvb0q6mLT1h17Z16CJSBPB5AO8AcCWA94rIle2qv8p9AG5yf7sTwM4QwjYAO6vry80sgI+FEF4H4DoAH6pei060ZUnQrjXkwra0aw29YdcQQlt+AFwP4Htq/S4Ad7WrflXvVgBPqvWnAWyqLm8C8HQH2vQAgBu6oS20K21Lu/auXdspuZwP4CW1fqD6t05zbgjhIABUf29sZ+UishXA1QB+0um2NAntmkGP25Z2zaCb7drODn2hAM8r2mdSRFYB+AaAO0IIpzrdniahXRcgB7alXReg2+3azg79AIAtav0CAK+0sf4sDovIJgCo/j7SjkpFpIS5G+PLIYRvdrItS4R2deTEtrSroxfs2s4O/REA20TkYhHpB/AeAA+2sf4sHgRwa3X5VsxpY8uKiAiAewHsDiF8ppNtaQG0qyJHtqVdFT1j1zZ/SLgZwDMA9gH4ZAc+ZHwFwEEAM5h7A7kNwHrMfZ3eW/29rg3t+A3MDV8fB/Bo9efmTrSFdqVtadf82JVT/wkhJCdwpighhOQEduiEEJIT2KETQkhOYIdOCCE5gR06IYTkBHbohBCSE9ihE0JITvh/iWvRzNmTR8sAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 对 train_data 进行归一化操作\n",
    "mean = np.mean(train_data, axis=0)\n",
    "std = np.std(train_data, axis=0)\n",
    "train_data = (train_data - mean) / (std + 1e-7)\n",
    "\n",
    "# 观察归一化后的图片\n",
    "plt.figure\n",
    "plt.subplot(1,3,1)\n",
    "plt.imshow(train_data[26].reshape((28,28)))\n",
    "plt.title('label: %s' % train_labels[0][26])\n",
    "plt.subplot(1,3,2)\n",
    "plt.imshow(train_data[16].reshape((28,28)))\n",
    "plt.title('label: %s' % train_labels[0][16])\n",
    "plt.subplot(1,3,3)\n",
    "plt.imshow(train_data[5].reshape((28,28)))\n",
    "plt.title('label: %s' % train_labels[0][5])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 修正标签\n",
    "由于我们只分类 0 和 非0，所以我们不妨把非零的标签都转换为 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "1\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "train_labels[train_labels!=0] = 1\n",
    "val_labels[val_labels!=0] = 1\n",
    "test_labels[test_labels!=0] = 1 \n",
    "\n",
    "print(train_labels[0][26])\n",
    "print(train_labels[0][16])\n",
    "print(train_labels[0][5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 估计 Lipschitz 常数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lipschitz Constant:\t2347716.5162674896\n"
     ]
    }
   ],
   "source": [
    "# 因为 dd L 和 w 有关，所以我们放缩 w 项，估算一个较大的 Lipschitz 常数.\n",
    "Lipschitz = np.max(np.linalg.eigvals(np.dot(train_data.T,train_data)))\n",
    "print('Lipschitz Constant:\\t%s' % Lipschitz)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 进行梯度下降计算参数\n",
    "分别使用 GD 和 SGD 进行参数学习。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def CrossEntropyLoss(P, y):\n",
    "    '''\n",
    "    Computing CrossEntropyLoss.\n",
    "    \n",
    "    Input:\n",
    "    - P: the probability. N x C\n",
    "    - y: the ground truth label. N x 1\n",
    "    Return:\n",
    "    - Loss: the Cross Entropy Loss. \n",
    "    - dL: the gradient of back propagation gradient.  N x C\n",
    "    '''\n",
    "    N, C = P.shape\n",
    "    # 多分类交叉熵\n",
    "    L = np.zeros((N, C))\n",
    "    for i in range(N):\n",
    "        L[i][y[i]] = -np.log(P[i][y[i]])\n",
    "    dL = np.zeros((N, C))\n",
    "    for i in range(N):\n",
    "        dL[i][y[i]] = - 1 / (P[i][y[i]] * N)\n",
    "    Loss = np.sum(L) / N\n",
    "    return Loss, dL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expected Answer:\t0.916290731874155\t[[0, 0, -1.25], [0, -1.25, 0]]\n",
      " Result:\t0.916290731874155\t[[ 0.    0.   -1.25]\n",
      " [ 0.   -1.25  0.  ]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 使用这个模块来测试上面的交叉熵是否正确\n",
    "P = np.array([[0.3,0.3,0.4],[0.3,0.4,0.3]])\n",
    "y = np.array([2,1])\n",
    "Loss, dL = CrossEntropyLoss(P, y)\n",
    "Expected_Loss = (-np.log(0.4)-np.log(0.4)) / 2\n",
    "Expected_dL = [[0,0,-1.25],[0,-1.25,0]]\n",
    "print(\n",
    "    'Expected Answer:\\t%s\\t%s\\n' % (Expected_Loss, Expected_dL),\n",
    "    'Result:\\t%s\\t%s\\n' % (Loss,dL)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Affine 层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Affine_forward(X, w, b):\n",
    "    '''\n",
    "    Affine forward.\n",
    "    \n",
    "    Input:\n",
    "    - X: input data. N x D.\n",
    "    - w: weight. D x C\n",
    "    - b: bias. C x 1\n",
    "    \n",
    "    Return:\n",
    "    - out: the output of Affine layer. N x C\n",
    "    - cache: the variable that used to back prop.\n",
    "    '''\n",
    "    out = np.dot(X, w) + b\n",
    "    cache = (X, w, b)\n",
    "    return out, cache"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Affine_backward(dout, cache):\n",
    "    '''\n",
    "    Affine backward.\n",
    "    \n",
    "    Input: \n",
    "    - dout: the gradient from upstream. N x C\n",
    "    Return:\n",
    "    - dw: the gradient of w. D x C\n",
    "    - db: the gradient of b. C x 1\n",
    "    - dX: the gradient of X. N x D\n",
    "    '''\n",
    "    X, w, b = cache\n",
    "    N = X.shape[0]\n",
    "    \n",
    "    db = np.sum(dout,axis=0) / N\n",
    "    dw = np.dot(X.T, dout) / N\n",
    "    dX = np.dot(dout, w.T) / N\n",
    "    \n",
    "    return dw, db, dX"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Softmax 激活函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Softmax_forward(X):\n",
    "    '''\n",
    "    Compute the softmax forward.\n",
    "    Input:\n",
    "    - X: the scores. N x C\n",
    "    \n",
    "    Return:\n",
    "    - prob: the probability. N x C\n",
    "    - cache: the variable that used for back prop.\n",
    "    '''\n",
    "    (N, C) = X.shape\n",
    "    # 修正 X, 防止指数爆炸\n",
    "    X = X - np.max(X,axis=1).reshape(N,1).repeat(C,axis=1)\n",
    "    prop = np.exp(X) / np.sum(np.exp(X),axis=1).reshape(N,1) \n",
    "    cache = prop\n",
    "    return prop, cache"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expected Answer:\t[0.3333333333333333, 0.3333333333333333, 0.3333333333333333]\n",
      " Result:\t[[0.33333333 0.33333333 0.33333333]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 测试 \n",
    "a = np.array([[1,1,1]])\n",
    "prop,_ = Softmax_forward(a)\n",
    "print(\n",
    "    'Expected Answer:\\t%s\\n' % [1/3,1/3,1/3],\n",
    "    'Result:\\t%s\\n' % prop\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Softmax_backward(dout, cache):\n",
    "    '''\n",
    "    Compute the back prop gradient.\n",
    "    \n",
    "    Input:\n",
    "    - dout: the gradient comes from upstream. N x C\n",
    "    - cache: the variable that used to compute the gradient.\n",
    "    \n",
    "    Retuen:\n",
    "    - dscore: the gradient of the scores. N x C\n",
    "    '''\n",
    "    \n",
    "    p = cache # N x C\n",
    "    (N, C) = np.shape(p)\n",
    "    # 计算比较复杂, 是一个比较复杂的张量, 我也不知道怎么向量化表示\n",
    "    #dp = np.zeros((N, C, N, C))\n",
    "#     for i in range(N):\n",
    "#         for j in range(C):\n",
    "#             for m in range(N):\n",
    "#                 for n in range(C):\n",
    "#                     dp[i][j][m][n] = p[i][j] * (i==m) * (j==n) - p[i][j] * p[m][n] * (i==m)\n",
    "    # dp = np.einsum('ij,imjn->ijmn',p,np.einsum('im,jn->imjn',np.eye(N),np.eye(C))) - np.einsum('ijmn,in->ijmn',np.einsum('ij,mn->ijmn',p,np.ones((N,C))), p)\n",
    "    dp = np.einsum('ij,imjn->ijmn',p,np.einsum('im,jn->imjn',np.eye(N),np.eye(C),optimize=True),optimize=True) - np.einsum('ijmn,im->ijmn',np.einsum('ij,mn->ijmn',p,p,optimize=True), np.eye(N),optimize=True)\n",
    "    # 使用 einstein 求和约定计算张量，dot 真要命\n",
    "    dscore = np.einsum('ij,ijkl->kl',dout,dp)\n",
    "    # dscore = np.einsum('ijkl,kl->ij',dp,dout)\n",
    "    # dscore = np.einsum('ijkl,ij->kl',dp,dout)\n",
    "    return dscore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expected Answer:\t[0.5, -0.5]\n",
      "Result:\t[[ 0.5 -0.5]]\n"
     ]
    }
   ],
   "source": [
    "# 测试\n",
    "dout = np.array([[1,-1]])\n",
    "p = np.array([[0.5,0.5]])\n",
    "print('Expected Answer:\\t%s' % [0.5,-0.5])\n",
    "print('Result:\\t%s' % Softmax_backward(dout,p))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression 模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LogisticRegression(object):\n",
    "    def __init__(self, input_dim, num_classes, lr=1e-3, batch=50, weight_scale=1e-2, weight_decay=1.0, reg=0.0):\n",
    "        '''\n",
    "        Initial the network.\n",
    "        \n",
    "        Input:\n",
    "        - input_dim: input dimension.\n",
    "        - num_classes: the prediction classes.\n",
    "        - lr: learning rate.\n",
    "        - batch: mini batch that used for SGD.\n",
    "        - weight_scale: weight scale.\n",
    "        - b: bias.\n",
    "        - reg: regularization parameter.\n",
    "        '''\n",
    "        self.w = weight_scale * np.random.randn(input_dim, num_classes)\n",
    "        self.b = np.zeros(num_classes)\n",
    "        self.lr = lr\n",
    "        self.batch = batch\n",
    "        self.weight_decay = weight_decay\n",
    "        self.reg = reg\n",
    "        self.Loss = []\n",
    "        self.Acc = []\n",
    "    \n",
    "    def train(self, data, labels):\n",
    "        '''\n",
    "        Compute the prediction. \n",
    "        \n",
    "        Input:\n",
    "        - data: the input data.\n",
    "        - labels: the labels of the data.\n",
    "        '''\n",
    "        acc = 0\n",
    "        iteration = 0\n",
    "        while acc < 0.97:\n",
    "        #while iteration < 200:\n",
    "            index = random.sample(range(len(data)), self.batch)\n",
    "            mini_data = train_data[index]\n",
    "            mini_labels = train_labels[0][index]\n",
    "            # forward pass\n",
    "            scores, cache = Affine_forward(mini_data, self.w, self.b)\n",
    "            prop, cache2 = Softmax_forward(scores)\n",
    "            Loss, dL = CrossEntropyLoss(prop, mini_labels)\n",
    "            \n",
    "            # Compute prediction and accuracy\n",
    "            pred_y = np.argmax(prop,axis=1)\n",
    "            acc = np.sum((pred_y == mini_labels))/len(mini_labels)\n",
    "            self.Loss.append(Loss)\n",
    "            self.Acc.append(acc)\n",
    "            \n",
    "                        \n",
    "            if np.mod(iteration,1) == 0:\n",
    "                print('iteration:\\t%s\\t\\t\\tLoss:\\t%s'% (iteration, Loss))\n",
    "                print('Accuracy:\\t%s' % acc)\n",
    "            \n",
    "            # backward pass\n",
    "            dscores = Softmax_backward(dL, cache2)\n",
    "            dw, db, _ = Affine_backward(dscores, cache)\n",
    "            \n",
    "            # update\n",
    "            iteration = iteration + 1\n",
    "            self.w = self.w - self.lr * dw\n",
    "            self.b = self.b - self.lr * db\n",
    "        \n",
    "    def test(self, data):\n",
    "        '''\n",
    "        Test Stage. This part is useless for this homework, I'll finish this if have time rest.\n",
    "        '''\n",
    "        scores, cache = Affine_forward(data, self.w, self.b)\n",
    "        prop, cache2 = Softmax_forward(scores)\n",
    "        \n",
    "        pred_y = np.argmax(prop,axis=1)\n",
    "        return pred_y\n",
    "    \n",
    "    def history(self):\n",
    "        '''\n",
    "        Return the log.\n",
    "        '''\n",
    "        return self.Loss, self.Acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 在小数据集上过拟合\n",
    "现在小数据集上验证算法是否正确，之后再在大数据集上运行。在小数据集上过拟合使在小数据集上 acc > 0.97。因为第一个 Loss 和相应的权重初始化有关系，所以 Loss 曲线会出现一个峰。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iteration:\t0\t\t\tLoss:\t1.0294037215347691\n",
      "Accuracy:\t0.415\n",
      "iteration:\t1\t\t\tLoss:\t10.87601853880798\n",
      "Accuracy:\t0.82\n",
      "iteration:\t2\t\t\tLoss:\t3.01043259544103\n",
      "Accuracy:\t0.9\n",
      "iteration:\t3\t\t\tLoss:\t1.1005886036638328\n",
      "Accuracy:\t0.935\n",
      "iteration:\t4\t\t\tLoss:\t0.5627034610924008\n",
      "Accuracy:\t0.945\n",
      "iteration:\t5\t\t\tLoss:\t0.15203695385222163\n",
      "Accuracy:\t0.975\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VPW9//HXJ8lM9oUsrAmEHWTHEBS0tVZ73SpW+6NuIIJLF61etV672VZva5er12rtbVUQlKKl2lZabV2qVkUFElbZwx62JJA9ZJ3P7485xLAmwExOZubzfDzmMXNmzpz5TMT3fOd7zvmMqCrGGGPCS5TbBRhjjAk8C3djjAlDFu7GGBOGLNyNMSYMWbgbY0wYsnA3xpgwZOFujDFhyMLdhD0R2S4iF7ldhzGdycLdGGPCkIW7iVgicquIFInIQRFZJCK9nftFRP5XREpEpFJEVovISOexy0RknYhUi8huEbnP3XdhzPFZuJuIJCIXAo8AU4FewA7gJefhLwGfA4YAacDXgAPOY7OB21U1GRgJvNOJZRvTYTFuF2CMS24A5qjqcgAR+S5QLiK5QBOQDAwDlqrq+jbPawLOEpFVqloOlHdq1cZ0kI3cTaTqjX+0DoCq1uAfnfdR1XeA3wBPAftF5GkRSXFWvQa4DNghIv8WkXM7uW5jOsTC3USqPUC/wwsikghkALsBVPUJVT0bGIF/euY7zv3LVHUK0B34K7Cwk+s2pkMs3E2k8IhI3OEL/lC+WUTGikgs8DNgiapuF5EJIjJRRDxALVAPtIiIV0RuEJFUVW0CqoAW196RMSdh4W4ixevAoTaX84EfAq8Ae4GBwLXOuinAM/jn03fgn675H+exacB2EakCvg7c2En1G3NKxH6swxhjwo+N3I0xJgxZuBtjTBiycDfGmDBk4W6MMWHItTNUMzMzNTc3162XN8aYkFRYWFimqlntredauOfm5lJQUODWyxtjTEgSkR3tr2XTMsYYE5Ys3I0xJgxZuIeAZz/YyoznllJV3+R2KcaYEGHh3sUtLirjp6+v572Npdwyr4D6JmtlYoxpn4V7F1ZW08Ddf1zJgMxEfnHNKJZtP8gdC5bT1OJzuzRjTBdn4d5F+XzKPQtXUXmoid9cP56vTejLQ1NG8vb6Ev7r5dX4fNYTyBhzYvZLTF3U0x9s5f1Npfz3VSMZ3sv/OxHTzulHeW0jj721ibQELz+8Yjgi4nKlxpiu6JRH7iIyx/nh4E/b3JcuIm+JyGbnultgy4wsy3eW8z9vbOSyUT25YWLfIx6788JBzJiUy5zF23jq3SKXKjTGdHWnMy0zF7jkqPseAP6lqoOBfznL5jRU1jVx54IV9EyN45GrRx8zMhcRHrziLL4yrg//8+Ym5n/SofMZjDER5pTDXVXfBw4edfcUYJ5zex5w1RnWFZFUlf96ZTX7q+r5zfXjSY33HHe9qCjhl18dzReHdeeHr37K31bt6eRKjTFdXaB2qPZQ1b0AznX3460kIreJSIGIFJSWlgbopcPH/E928M+1+7j/kqGMzUk76bqe6CieumE8E/qlc8/Clfx7k/09jTGf6dSjZVT1aVXNU9W8rKx2+95ElLV7Knn4tfVcMDSLW84b0KHnxHmieeamPAZ1T+brLxRSuKM8yFUaY0JFoMJ9v4j0AnCuSwK03YhQ29DMnS+uIC3ew6P/bwxRUR0/AiY13sPzM/PpkRLLzLnL2LivOoiVGmNCRaDCfRFwk3P7JuDVAG03Ijz46lq2ldXy+LVjyUiKPeXnZyXH8sKsicR5opg2ewm7DtYFoUpjTCg5nUMhXwQ+BoaKSLGIzAJ+DlwsIpuBi51l0wF/Xl7MK8uLufPCwUwamHna28lJT+D5mRNpaPZx4+wllFY3BLBKY0yoOZ2jZa5T1V6q6lHVbFWdraoHVPWLqjrYuT76aBpzHFtKa/jBXz8lv386375w0Blvb2jPZObMmEBJVQPT5yyl8pA1GjMmUln7AZfUN7Vwx4IVxMZE8cS144iJDsx/irP7deN3086mqKSaW+Yt41CjNRozJhJZuLvkZ6+vZ/3eKh6dOoaeqXEB3fbnh2Tx2NSxFOwot0ZjxkQoC3cX/PPTvTz/8Q5uOa8/Fw7rEZTX+PKY3jw8ZST/2lDC/dZozJiIY43DOtmug3Xc//JqRmencv8lw4L6Wjee04+Kukb+581NpMZ7+NGXz7JGY8ZECAv3TtTU4uOul1bgU3jyunF4Y4L/xelbXxhEeV0Tsz/cRnqil29/cXDQX9MY4z4L90702FubWL6zgievG0e/jMROeU0R4fuXDaeironH3tpEtwQP087N7ZTXNsa4x8K9k7y/qZT/e28L1+Xn8OUxvTv1taOihF9cM4rKQ008uGgtKfEepozt06k1GGM6l+1Q7QQl1fXcs3AlQ3ok8eAVI1ypISY6it9cP44Juencu3AV7220DhHGhDML9yBr8Sn/+ceV1DQ089T144n3RrtWS5wnmmdvymNoz2S+Pr+Qwh12rpkx4crCPcj+770iFhcd4CdXjmBwj2S3yyElzsO8mfn0So3n5ueWsWFfldslGWOCwMI9iJZtP8hjb23iyjG9mZqX43Y5rTKTYnl+Zj4J3himzV7KzgPWaMyYcGPhHiTltY18+8UV5KQn8NOvjOxyx5fnpCfwwqx8mlr8jcZKquvdLskYE0AW7kGgqnzn5dWU1TTw5HXjSI47/s/luW1wj2SemzGBspoGps+2RmPGhBML9yCY+9F23l6/nwcuHc7o7JP/XJ7bxvXtxu+nnc2W0hpmzbVGY8aECwv3APt0dyWPvL6Bi4Z3Z+bkXLfL6ZDzB2fx+NfGUbiznG/+odAajRkTBizcA6imoZk7FiwnI8nLr746psvNs5/M5aN78dOrRvHuxlLu+9MqazRmTIizM1QDRFX5/l/WsPNgHS/ddi7dEr1ul3TKrp/Yl/K6Rn71xkbS4j38+MoRIfUBZYz5jIV7gPypoJhXV+7h3ouHkN8/3e1yTts3LxhIRV0jz3ywjW6JXu6+aIjbJRljToOFewBs3l/Ng4s+ZdLADL75hTP/uTw3iQjfu2w45XVNPP72ZroleLlpUq7bZRljTpGF+xk6/HN5id4YHv/aWKKjQn8aQ0T4+dX+RmM/WrSWtARrNGZMqLEdqmfoJ39bx8b91Tw6dQzdUwL7c3luiomO4snrxnHOAH+jsXc3WKMxY0KJhfsZ+PvqPby4dCe3f34AFwzt7nY5ARfnieaZ6XkM65XMN/5QyLLt1mjMmFBh4X6adh6o47uvrGFc3zTu+9JQt8sJmuQ4D3Nvzqd3ajwz5y5j/V5rNGZMKLBwPw2NzT7ufHE5IvDEtePwRIf3nzEzKZbnZ+WTFBvD9DlL2XGg1u2SjDHtCO9UCpJfvbGBVcWV/PKro8lJT3C7nE6R3c3faKy5xce02UspqbJGY8Z0ZRbup+idDft55oNtTDunH5eM7OV2OZ1qUPdknrs5399obM5SKuus0ZgxXZWF+ynYV1nPvQtXMbxXCt+/fLjb5bhibE4aT0/LY2tpLTPnLaOusdntkowxx2Hh3kEtPuWul1bQ0OzjN9ePI87j3s/lue28wZn8+tqxrNhZzjfmL6ex2RqNGdPVWLh30BP/2sySbQd5eMpIBmYluV2O6y4d1YuffWUU/95kjcaM6YrsDNUO+HjLAZ58ZzNXj+vDNWdnu11Ol3Ftfl/K65r4xT83kJbg4SfWaMyYLsPCvR0Hahq4+48ryM1I5OGrRrpdTpfzDafR2O/f30pagpd7LrZGY8Z0BRbuJ+HzKff9aRXldU3MmTGBxFj7cx3PA5cOo7yukSf+tZluCR5untzf7ZKMiXiWVicx+8NtvLuxlIemjGBE71S3y+myRISffcXfaOwnf1tHWoKHr4yz6Stj3BTQHaoisl1E1ojIShEpCOS2O9vKXRX84p8b+I8RPZh2Tj+3y+nyYqKj+PW145g0MIP7/rSadzbsd7skYyJaMI6W+YKqjlXVvCBsu1NU1Tdx54vL6ZESxy+vCa2fy3NTnCeap6fnMaJ3Ct+Yv5yl26zRmDFusUMhj6KqfPeVNeypqOeJ68aRmuBxu6SQkhQbw3MzJtCnWzyz5i1j3R5rNGaMGwId7gq8KSKFInLb0Q+KyG0iUiAiBaWlpQF+6cBYsHQnr63Zy31fGsrZ/bq5XU5IykiKZf6siSQ7jca2l1mjMWM6W6DDfbKqjgcuBb4lIp9r+6CqPq2qeaqal5WVFeCXPnMb9lXx0N/Wcf7gTG7/3AC3ywlpvdPieX7WRHyq3Dh7Cfut0ZgxnSqg4a6qe5zrEuAvQH4gtx9MdY3N3LFgBSnxHh6bOpaoMPi5PLcN6p7E3JsnUF7byPTZS6moa3S7JGMiRsDCXUQSRST58G3gS8Cngdp+sP140Vq2lNbw+NfGkpUc63Y5YWN0dhrPTM9jW1ktM+daozFjOksgR+49gA9FZBWwFHhNVf8ZwO0Hzasrd7OwoJhvXTCIyYMy3S4n7EwalMkT141j5a4Kvm6NxozpFAELd1XdqqpjnMsIVf1poLYdTNvKavnen9cwIbcbd1802O1ywtYlI3vy86tH8/6mUu5ZuJIWazRmTFBF9BmqDc0t3Pnicjwx/hNwYsL85/LcNnVCDuV1jTzyjw2kxnv476tG2jkExgRJRIf7I69v4NPdVTwzPY/eafFulxMRbv/8QMrrmvjdv7eQnujl3jD+cXFj3BSx4f7m2n3M/Wg7N0/O5eKzerhdTkT5r0uGUlHXyJPvFJGW4GXWedZozJhAi8hw311xiO+8vJqRfVJ44NJhbpcTcUSEn35lFBV1TTz893V0S/Bw9XhrNGZMIEXcJHNzi4+7XlxBc4uPJ68bT2xM5P5cnpuio4RfXzeWyYMy+M7Lq3l7nTUaMyaQIi7cH397MwU7yvnZ1aPon5nodjkRLTYmmt9Py2Nk7xS+tWA5S7YecLskY8JGRIX7h5vLeOq9IqbmZTNlbB+3yzE4jcZuzicnPYFb5hXw6e5Kt0syJixETLiXVjdw9x9XMjAriR9fOcLtckwb6YleXpiVT0q8hxnPLWWbNRoz5oxFRLj7fMo9C1dSXd/EU9ePJ8EbkfuRu7ReqfE8PysfVbjx2SXsq7RGY8aciYgI99+9v4UPNpfxoy+PYGjPZLfLMScwMCuJuTfnU3moiWmzl1ijMWPOQNiHe+GOgzz65iYuH92L6/Jz3C7HtGNUdirPTM9jx8E6Lnrs39z90gr+VLCLPRWH3C7NmJAS1vMTFXWNfPvFlfROi+ORq0fZqe4h4tyBGbwwM5/5S3byweYy/rpyDwD9MxOZPCiDyQMzOXdgBmkJXpcrNabrCttwV1Xuf3k1+6vqefkbk0iJs5/LCyUTB2QwcUAGPp+ycX81i4vK+GjLAf6yfDfzP9mJCIzoncLkQZlMHpjJhNx04r12zoIxh4mqO9358vLytKCgIGjbf/7j7Tz46lq+f9lwbrVfVQobTS0+Vu2qYHHRARZvKWPFznKaWhRvdBTj+6UxeWAmkwZlMiY71RrBmbAkIoWqmtfueuEY7mv3VPKVpz5i8qAMZt80wX5VKYzVNTazdNtBPtpygMVFZax1fpA7KTaGcwakM2lgJpMHZTKkR5JNy5mw0NFwD7tpmdqGZu5csIJuiR4etZ/LC3sJ3hguGNqdC4Z2B+BgbSMfb/GP6hcXlfH2+hIAspJjmTQwwxnZZ5DdLcHNso0JurAL9x/+9VO2H6hlwa3nkJ5oO9wiTXqil8tH9+Ly0b0AKC6v46Oiw2F/gFednbO5GQlMcubrzx2YYf9WTNgJq3B/ubCYP6/Yzd0XDeacARlul2O6gOxuCUydkMDUCTmoKpv217C4yD+qX7RyDwuW+HfOntXLv3N20sAM8vun24luJuSFzZx7UUkNX37yQ8bkpPKHW84h2qZjTDuaWnysLq7ko6IyPiwqY8XOChpbfHiihXF9uzF5YCbnDc5gdHYaHts5a7qIiNqhWt/UwlVPLaakuoF/3HU+PVLiArJdE1kONbawbPtB/8h+i3/nrCokeqOZOCCDSQMzOG9wJkN7JNvOWeOaiNqh+t+vrWPDvmqemzHBgt2ctnhvNJ8bksXnhmQBUF7byCdbD/Chc4z9Oxv8O2czk7ycOzCT8wZlMGlgJjnptnPWdD0hH+7/WLOX+Z/s5Nbz+/OFYd3dLseEkW6JXi4d1YtLR/l3zu6uOOQ/maqojMVbDvC3Vf6ds33TE/xnzg7K5NwBGWQkxbpZtjFAiE/L7DpYx2VPfMCArCT+dPu5eGNsXtR0DlWlqKSGD4v8R+Es2XqA6oZmAIb3SvGP6gdlkp+bTmJsyI+hTBcS9nPuTS0+pv7+Y4r21/Dat8+nb4Z9NTbuaW7xsXq3f+fs4qIDFO4op7HFR0yUMK5vmr9NwqBMxubYzllzZsI+3B/5x3p+/++tPHX9+NZjmo3pKg41tlCw4yCLiw7w0ZYy1uyuRBUSvNFM7J/uHHaZybCeyXainTklYb1D9b2NJfz+31u5fmJfC3bTJcV7ozl/cBbnD/bvnK2o8++cPdwT593X1gOQkejl3IEZrQ3Q7BuoCZSQG7nvr6rnsl9/QFZyLH/91mTiPNYJ0ISevZWH/KN65xj7kuoGAOI90XRL8JCW4KVbonOd4KFbgveI26nOdbcEDylxHhv9R5CwHbkvWLKTusYWfnP9OAt2E7J6pcbz1bOz+erZ2agqW0pr+GjLAXYeqKO8romKukbK6xrZW1FFeV0jlYea8J1gHBYlkBp/+APgqA+CxLb3Hf5A8N+2/3/CW8iN3FWVzSU1DOlhP5dnIofPp1TVN1Fe10R5XaM//GsP3z722v/h0MShppYTbrOj3xLafigkx8XYtwSXhe3IXUQs2E3EiYoS0pzQ7U9ih59X39TSGvrHfBDUNtq3hDAWcuFujOm4OE80PVOj6Zna8TO3T/YtofLQ4Q8K/4fC3sp61u+tOqNvCSlxHpLiYkiOiyEpNobkOA/JbZYTvfZt4XRYuBtjjtBVviUcJgJJXifs23wAJMXFkNJ2OTam9UOh7XJSXAzJsR7iPFER1RPIwt0YExCn+y2htrGZmoZmqusPX5pal2uc5eq2yw3+D4ld5XWt953sW8NhMVHS5huC8+2g7QeA84GQ0voh4mn99tD220WonIQWsHAXkUuAXwPRwLOq+vNAbdsYE56iosSZhvHQK/X0t9PU4qP2BB8Q/g+GJueD4vAHSRPV9c3sraxnc8ln9zW1tH+ASWxM1DFTR0d8YBz17eKzD5HPlhO9MUFvSx6QcBeRaOAp4GKgGFgmIotUdV0gtm+MMSfjiY5qnUo6XapKQ7PviA+Amvpmqo5arm449gPkQFld6zeMmoZm2jsI8aEpI5h+bu5p19oRgRq55wNFqroVQEReAqYAFu7GmJAgIsR5oonzRJOVfPqdPX0+pa6p5fhTSs4HQl6/9ABWfnyBCvc+wK42y8XAxKNXEpHbgNsA+vbtG6CXNsaYriMqSkiK9U/NnMr+h4DXEaDtHG/y6JgvJqr6tKrmqWpeVlZWgF7aGGPM0QIV7sVATpvlbGBPgLZtjDHmFAWk/YCIxACbgC8Cu4FlwPWquvYkzykFdpzmS2YCZaf53FBl7zky2HuODGfynvupartTHwGZc1fVZhG5A3gD/6GQc04W7M5zTnteRkQKOtJbIZzYe44M9p4jQ2e854Ad566qrwOvB2p7xhhjTl9onGpljDHmlIRquD/tdgEusPccGew9R4agv2fX+rkbY4wJnlAduZsIJiLviUi5iJz+aYTGhDkLdxNSRCQXOB//SXJXduLrWgdVE1JCLtxF5BIR2SgiRSLygNv1BJuIzBGREhH51O1aOouI5IjIuyKyXkTWishdbR6eDnwCzAVuavOceBF5VER2iEiliHwoIvHOY+eJyEciUiEiu0RkhnP/eyJyS5ttzBCRD9ssq4h8S0Q2A5ud+37tbKNKRApF5Pw260eLyPdEZIuIVDuP54jIUyLy6FHv8W8icneb5TgRWSoiq5z3/JOA/DG7OOdvtkJE/u52LZ1BRLaLyBoRWSkip/47o6dCVUPmgv8Y+i3AAMALrALOcruuIL/nzwHjgU/drqUT33MvYLxzOxn/CXJnOctFwDeBs4EmoIdz/1PAe/j7HEUDk4BYoC9QDVwHeIAMYKzznPeAW9q87gzgwzbLCrwFpAPxzn03OtuIAe4F9gFxzmPfAdYAQ/G35BjjrJuP/4ztKGe9TKDucO3OfQIkObc9wBLgHLf/W3TCf+t7gAXA392upZPe73YgszNeK9RG7q3dJ1W1ETjcfTJsqer7wEG36+hMqrpXVZc7t6uB9UAfETkP6AcsVNVC/B/014tIFDATuEtVd6tqi6p+pKoNwA3A26r6oqo2qeoBVV15CuU8oqoHVfWQU898ZxvNqvoo/g+Qoc66twA/UNWN6rfKWXcpUIn/DG6Aa4H3VHV/m/esqlrjLHqcS1gf7SAi2cDlwLNu1xKOQi3cj9d9so9LtZhO4Myxj8M/kr0JeFNVD5+2vcC5LxOIwx/2R8s5wf0d1fbfGyJyrzNdVCkiFUCq8/rtvdY8/KN+nOsXjl7BmaJYCZQAb6nqkjOoOxQ8DtwP+NwupBMp8KYzZXdbMF8o1HYSdaj7pAkPIpIEvALcjX8KZioQLSL7nFVigTT80zj1wED8U3Vt7cL/je94aoGENss9j7NO678vZ379v/CPwNeqqk9Eyvns3+Uup4bj7R+ZD3wqImOA4cBfj3kh1RZgrIikAX8RkZGqGpb7WkTkCqBEVQtF5AK36+lEk1V1j4h0B94SkQ3Ot/OAC7WRu3WfjBAi4sEf7H9Q1T8DVwEtwFnAWOcyHPgA/07WOcBjItLbGQGf6xwq+QfgIhGZKiIxIpIhImOdl1kJXC0iCSIyCJjVTlnJQDNQCsSIyINASpvHnwUeFpHB4jdaRDIAVLUYf0O9F4BXDk/zHI+qVuDfH3BJR/5WIWoycKWIbMc/vXqhiMx3t6TgU9U9znUJ8BdOPPA4Y6EW7suAwSLSX0S8+OcuF7lckwkw8f9E/Wxgvao+5tx9E/Ccqu5U1X2HL8Bv8M+rP4B/Z+Yy/PsofoF/B+ZO4DL8Oz8P4g/0Mc42/xdoBPbjnzb5QzulvQH8A/8O3h34vy20nbZ5DFgIvAlUOe8hvs3j84BRHH9KJssZseMc5XMRsKGdekKWqn5XVbNVNRf//8fvqOqN7TwtpIlIoogkH74NfInjf8sLzOs5e3BDhohchn+u7nD3yZ+6XFJQiciLwAX453X3Az9S1dmuFhVkzo7TD/CH9eH52O+pvzldyBKRz+GfnslVVd9Rj43GH/7R+AddC1X1oc6vsvM50zL3qeoVbtcSTCIyAP9oHfxT4guCmV8hF+7GhCJnmuklYFWkhLZxV7vTMu2dROPMLT7hnFS0WkTGB75MY0KXiAwHKvDv+H3c5XJMhOjInPtcTr5j51JgsHO5Dfi/My/LmPChqutVNVFVJ6lqldv1mMjQbrh34CSaKcDzzkkYnwBpItIrUAUaY4w5dYE4zv1EJxbtPXpF56D92wASExPPHjZsWABe3hhjIkdhYWGZdtJvqHb4xCJVfRqnSX1eXp4WFAS3b44xxoQbEdnRkfUCcZy7nVhkjDFdTCBG7ouAO0TkJWAiUKmqx0zJGGNMJFJVahtbKK9t5KBzGdQ9iZz0hPaffAbaDfe2J9GISDHwI/wd61DV3wGv4z8DsAh/G9Obg1WsMca4rbHZR3mdP6TLaxs5UNt4guUmf6DXNdLYfGRvtIenjGDaublBrbPdcFfV69p5XIFvBawiY4zpJD6fUnmoiYN1jUeMrD9bbuJgbQMH6/xBXV7bSHVD8wm3lxrvIT3RS7cED33S4hjVJ4VuiV7SE7x0S/SSkei/zs1IDPp7C7WukMYYc1yqSl1ji38EXdcmqNuMpA/WNlBe+1mYl9c14jvBSfpxnigyEmPpluihW4KX3IwE0tsEdXqbS7cEL2kJHjzRXaddl4W7MaZLamrxtU5r+Kc8nFF0bdMJwruRhubjt4aPjhK6JXhag3hw96TPRtIJTkC3GVmnJ3iJ90Z38jsOLAt3Y8wpU1UaW3w0NjuXFh9NzUpjSwsNzn1NLeo81uJc62frN7f4H2/x+adFjjN/XV1/4umP5LiY1iDulRrHWb1Tjghm/8jaQ3piLOkJXpLjYoiKOt5R2+HLwt2YLqxtiDa1DceWFhqbjwzYphafP1hbPltuG75HXLd5vOEU129s8dcSKN6YKDLaTHH0TU/4bLqjNaw9rVMkafFevDFdZ/qjq7JwN6YTNLX4OFDTSGl1A2U1DZRWN1Da5rrMua5taG4Tpv7wDiRPtOCNjsITE4U3OgpvjHOJPvI6MTam9bHY6Cg8bdeN8S/Htlnfc8y2BG90dJv1xVk/unW57fr+Fv4mkCzcjTlNLT7lYG3jMQFd1ia4Dwd5eV3TcbeRHBtDZnIsWUmxDOuZTHKs55iwjD0chtFReGOijwxLJzDbhmVs2+e3Dd/oqIibmohkFu7GtOHzKRWHmj4bXR9vpF3dQFlNIwdrG457pEWcJ4ruyXFkJnnpn5nIhNx0spJjyUqOJTPJf52V5L8d6jvtTNdl4W7CnqpSVd98RFAfE941DZRVN1JW00DzcRLbGx3lhLOX7G7xjOub1hrURwR2ciyJ3mibZjCus3A3Iau2ofmYkfVnUyKNRywffYYg+A+Py0zytgb08J4prVMkR4R2ciwpcTEW2CakWLibLqfFp2wtrWFbWW3riLq0pr51OuTwiPtQU8sxzxWBjERvazAPzEw8JqgP306L99gctAlbFu7GVarKvqp6Vu2qYOWuSlbtqmDN7kpqjjrFOy3B0zqiHpuTdpzA9o/A0xO8xHShswSNcYuFu+lUlYeaWFNcyariClbuqmDVrgpKqhsA/2F6w3ul8JVxfRiTk8aQHklkJceSkRhrxzUbc4os3E3QNDS3sH5vNaucEF9ZXMHW0trWxwdkJTJ5UCZjslMZk5PG8F4pxHns6BFjAsHC3QSEz6dsLav1B3mxP8zX7a1qPZPx8HTpOLIsAAAOMklEQVTKNeOzGZOdxqjsVFLjPS5XbUz4snA3p2V/VX3rtMqq4gpW76psbYWa6I1mdHYas84bwNgc/6i8Z0qcHW1iTCeycDftqqr/bJ7cP8VSyb6qegBiovzz5FPG9WZMdhpjc9IYkJVEtB2FYoyrLNzNERqaW9iwt/qIHZ5b2syT989M5JwB6YzJSWNMThpn2Ty5MV2ShXsE8/mUbQdq2+zwrGT9nqrWZlWZSV7G5qRx1Vj/0Sujs1NJS/C6XLUxpiMs3CNIyeF58mL/1Mqq4orWntkJ3mhG9Unl5sm5raPy3qk2T25MqLJwD1PV9U2s2V3pD3En0PdW+ufJo6OEYT2T+fKY3ozN9gf5oO42T25MOLFwDwONzT427qtmZesOzwqKSmtQp/9VbkYC+f3TGeME+YjeNk9uTLizcA8xqsr2A3XO6fr+EfnaPVWtjbEyEv3z5F8e09s/T94nlW6JNk9uTKSxcA8BOw/U8afCXazcVcHq4koqD/l/+CHeE82o7FRmTMp1RuWp9EmLt3lyY0zHwl1ELgF+DUQDz6rqz496vC8wD0hz1nlAVV8PcK0RaeeBOr76u484UNvI0B7JXDaqV+uJQYOykqxJljHmuNoNdxGJBp4CLgaKgWUiskhV17VZ7QfAQlX9PxE5C3gdyA1CvRFlf1U9N8z+hMYWH/+463yG9Eh2uyRjTIjoyLAvHyhS1a2q2gi8BEw5ah0FUpzbqcCewJUYmcprG7nx2SUcrGlk3s35FuzGmFPSkWmZPsCuNsvFwMSj1vkx8KaI3AkkAhcFpLoIVV3fxE3PLWXHwTrm3jyBMTlpbpdkjAkxHRm5H2/v3NE/MnkdMFdVs4HLgBdE5Jhti8htIlIgIgWlpaWnXm0EqG9q4dbnC1i7p4rfXj+eSQMz3S7JGBOCOhLuxUBOm+Vsjp12mQUsBFDVj4E44JhUUtWnVTVPVfOysrJOr+Iw1tTi444Fy1my7SCPTR3DRWf1cLskY0yI6ki4LwMGi0h/EfEC1wKLjlpnJ/BFABEZjj/cbWh+Cnw+5b4/reLt9SU8NGUkU8b2cbskY0wIazfcVbUZuAN4A1iP/6iYtSLykIhc6ax2L3CriKwCXgRmqOrRUzfmBFSVBxd9yqsr9/Cd/xjKtHP6uV2SMSbEdeg4d+eY9dePuu/BNrfXAZMDW1rk+NUbG5n/yU5u//wAvnnBQLfLMcaEATsDxmW/+/cWfvveFq7L78sDlwyzs0uNMQFh4e6iBUt28vN/bODLY3rz31eNtGA3xgSMhbtLFq3aw/f/uoYLh3XnsaljrN2uMSagLNxd8M6G/dzzx5VMyE3ntzeMx2P9YYwxAWap0sk+2XqAb8xfzvBeKcy+Kc/6qhtjgsLCvROtLq7glnkF5KQnMG9mPslxHrdLMsaEKQv3TrJ5fzU3zVlKWoKH+bMmkm4/oGGMCSIL906w62AdN85eQkx0FPNnTaRnapzbJRljwpyFe5CVVNVz4+wl1Df5eGFWPrmZiW6XZIyJABbuQVRR18i02UsprW7guZsnMKxnSvtPMsaYALBwD5LahmZmPLeMbWW1PDM9j/F9u7ldkjEmgtgPZAdBfVMLt71QwJrdlfz2hvFMHmQ92Y0xnctG7gHW3OLjzhdXsLjoAL/66mj+Y0RPt0syxkQgC/cA8vmU+19ezVvr9vOTK0dw9fhst0syxkQoC/cAUVV+8re1/HnFbu69eAg3Tcp1uyRjTASzcA+Qx97axLyPd3Dr+f2548JBbpdjjIlwFu4B8Mz7W3nynSK+lpfD9y4bbq17jTGus3A/Qy8t3clPX1/P5aN68bOrR1mwG2O6BAv3M/D31Xv47l/W8PkhWfzv18ZaT3ZjTJdh4X6a3ttYwn/+cSV5/brxuxvPxhtjf0pjTNdhiXQalm0/yNfnFzKkRzKzZ0wg3ms92Y0xXYuF+yn6dHclM59bRu+0eObNzCfFerIbY7ogC/dTUFRSw/Q5S0mJ9/dkz0yKdbskY4w5rg6Fu4hcIiIbRaRIRB44wTpTRWSdiKwVkQWBLdN9xeV1TJu9hCgR5t8ykd5p8W6XZIwxJ9Ru4zARiQaeAi4GioFlIrJIVde1WWcw8F1gsqqWi0j3YBXshtLqBm58dgm1Dc28dNu59Lee7MaYLq4jI/d8oEhVt6pqI/ASMOWodW4FnlLVcgBVLQlsme6prGti+pyl7K/y92Q/q7f1ZDfGdH0dCfc+wK42y8XOfW0NAYaIyGIR+URELglUgW6qa2zm5rlL2VJSw9PTz+bsfulul2SMMR3SkX7uxzszR4+zncHABUA28IGIjFTViiM2JHIbcBtA3759T7nYztTQ3MLtLxSyclcFv71hPOcPznK7JGOM6bCOjNyLgZw2y9nAnuOs86qqNqnqNmAj/rA/gqo+rap5qpqXldV1w7K5xcddL67kg81l/OKa0VwyspfbJRljzCnpSLgvAwaLSH8R8QLXAouOWuevwBcARCQT/zTN1kAW2ll8PuWBP6/hn2v38eAVZ/H/8nLaf5IxxnQx7Ya7qjYDdwBvAOuBhaq6VkQeEpErndXeAA6IyDrgXeA7qnogWEUHi6ry8GvreLmwmLsvGszM8/q7XZIxxpwWUT16+rxz5OXlaUFBgSuvfSKPv72Jx9/ezMzJ/fnhFda61xjT9YhIoarmtbeenaHqmP3hNh5/ezP/7+xsfnC5BbsxJrRZuAMLC3bx8N/XcenInjxy9SiirHWvMSbERXy4/2PNXh54ZTXnD87k8WvHEhMd8X8SY0wYiOgke39TKd9+aQXj+nbj99POJjbGWvcaY8JDxIZ74Y6D3P5CIYO6JzNnxgQSvB05n8sYY0JDRIb7uj1VzHhuGT1T43h+Zj6p8daT3RgTXiIu3LeW1jB9zhKSY2OYf8tEspKtJ7sxJvxEVLjvqTjEjc8uQRVeuGUifawnuzEmTEVMuJfVNHDj7CVU1zczb2Y+A7OS3C7JGGOCJiL2IlYeamL67KXsqTjEC7MmMrJPqtslGWNMUIX9yP1QYwuz5i5jc0k1v7vxbCbkWk92Y0z4C+twb2z2cfv8QpbvLOfxr43jgqFh9et/xhhzQmE7LdPiU/7zjyt5f1Mpv7xmNJePtp7sxpjIEZYjd1Xle39ew2tr9vKDy4czdYL1ZDfGRJawC3dV5aevreePBbv49oWDuOX8AW6XZIwxnS7swv037xTx7IfbmDEpl/+8eIjb5RhjjCvCKtznLt7Go29t4urxfXjwirOsJ7sxJmKFTbi/UljMj/+2ji+d1YNfXjPaerIbYyJaWIT7G2v3cf8rqzlvUCZPXj/OerIbYyJeyKfg4qIy7lywgtHZqdaT3RhjHCEd7st3lnPr8wUMyErkuRkTSIwN28P2jTHmlIRsuG/YV8WMOUvJSo7l+Vn5pCV43S7JGGO6jJAM9+1ltUybvZQEbwzzZ02ke3Kc2yUZY0yXEnLhvrfyEDc8u4QWnzL/lnxy0hPcLskYY7qcDoW7iFwiIhtFpEhEHjjJel8VERWRvMCVeKSXC4qpPNTEvJvzGdQ9OVgvY4wxIa3dPZAiEg08BVwMFAPLRGSRqq47ar1k4NvAkmAUetgdFw7iyrG96ZeRGMyXMcaYkNaRkXs+UKSqW1W1EXgJmHKc9R4GfgnUB7C+Y4iIBbsxxrSjI+HeB9jVZrnYua+ViIwDclT17yfbkIjcJiIFIlJQWlp6ysUaY4zpmI6E+/HO49fWB0WigP8F7m1vQ6r6tKrmqWpeVlZWx6s0xhhzSjpy1k8x0LYhejawp81yMjASeM9p1NUTWCQiV6pqwYk2WlhYWCYiO069ZAAygbLTfG6osvccGew9R4Yzec/9OrKSqOrJVxCJATYBXwR2A8uA61V17QnWfw+472TBfqZEpEBVg3ZETldk7zky2HuODJ3xntudllHVZuAO4A1gPbBQVdeKyEMicmUwizPGGHN6OtSMRVVfB14/6r4HT7DuBWdeljHGmDMRcmeoOp52uwAX2HuODPaeI0PQ33O7c+7GGGNCT6iO3I0xxpyEhbsxxoShkAv3jjYxCxciMkdESkTkU7dr6SwikiMi74rIehFZKyJ3uV1TsIlInIgsFZFVznv+ids1dQYRiRaRFSJy0rPbw4WIbBeRNSKyUkSCdrg4hNicu9PEbBNtmpgB1x3dxCyciMjngBrgeVUd6XY9nUFEegG9VHW505CuELgqzP87C5CoqjUi4gE+BO5S1U9cLi2oROQeIA9IUdUr3K4n2ERkO5CnqkE/aSvURu4dbWIWNlT1feCg23V0JlXdq6rLndvV+M+v6HPyZ4U29atxFj3OJXRGXqdBRLKBy4Fn3a4lHIVauLfbxMyEFxHJBcYR5FbSXYEzRbESKAHeUtVwf8+PA/cDPrcL6UQKvCkihSJyWzBfKNTC/aRNzEx4EZEk4BXgblWtcrueYFPVFlUdi79/U76IhO00nIhcAZSoaqHbtXSyyao6HrgU+JYz7RoUoRbu7TUxM2HCmXd+BfiDqv7Z7Xo6k6pWAO8Bl7hcSjBNBq505qBfAi4UkfnulhR8qrrHuS4B/oJ/qjkoQi3clwGDRaS/iHiBa4FFLtdkAszZuTgbWK+qj7ldT2cQkSwRSXNuxwMXARvcrSp4VPW7qpqtqrn4/z9+R1VvdLmsoBKRROcAAUQkEfgSELSj4EIq3E/UxMzdqoJLRF4EPgaGikixiMxyu6ZOMBmYhn80t9K5XOZ2UUHWC3hXRFbjH8S81d6P35iQ0wP4UERWAUuB11T1n8F6sZA6FNIYY0zHhNTI3RhjTMdYuBtjTBiycDfGmDBk4W6MMWHIwt0YY8KQhbsxxoQhC3djjAlD/x8GRZC3yFoe0gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 参数初始化\n",
    "input_dim = 784\n",
    "num_classes = 2\n",
    "mini_batch = 200\n",
    "model = LogisticRegression(input_dim, num_classes, lr=1000, batch=mini_batch, weight_scale=4e-2, weight_decay=1.0, reg=0.0)\n",
    "model.train(train_data[0:mini_batch], train_labels[0][0:mini_batch])\n",
    "\n",
    "# 画图\n",
    "Loss, Acc = model.history()\n",
    "plt.figure\n",
    "plt.subplot(2,1,1)\n",
    "plt.plot(Loss)\n",
    "plt.title('Loss')\n",
    "plt.subplot(2,1,2)\n",
    "plt.plot(Acc)\n",
    "plt.title('Accuracy')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用 GD 在全数据集上过拟合\n",
    "训练模型在全 training_data 上过拟合，使得在 training_data 上满足 acc > 0.97。因为电脑显存有限，不能够在 60000 的数据集上运行 GD 算法，所以我修改了数据集大小，改成在 大小为 8000 的数据集上进行 GD 和 SGD 的比较"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iteration:\t0\t\t\tLoss:\t0.8269287420125175\n",
      "Accuracy:\t0.553875\n",
      "iteration:\t1\t\t\tLoss:\t0.6019360752048041\n",
      "Accuracy:\t0.724125\n",
      "iteration:\t2\t\t\tLoss:\t0.5208808382280946\n",
      "Accuracy:\t0.779625\n",
      "iteration:\t3\t\t\tLoss:\t0.4728695836711729\n",
      "Accuracy:\t0.82075\n",
      "iteration:\t4\t\t\tLoss:\t0.43683789855699445\n",
      "Accuracy:\t0.846\n",
      "iteration:\t5\t\t\tLoss:\t0.40747606926990504\n",
      "Accuracy:\t0.864375\n",
      "iteration:\t6\t\t\tLoss:\t0.3825890491250804\n",
      "Accuracy:\t0.877875\n",
      "iteration:\t7\t\t\tLoss:\t0.3610735551584795\n",
      "Accuracy:\t0.890375\n",
      "iteration:\t8\t\t\tLoss:\t0.3421797136587069\n",
      "Accuracy:\t0.900375\n",
      "iteration:\t9\t\t\tLoss:\t0.32535486550203246\n",
      "Accuracy:\t0.907\n",
      "iteration:\t10\t\t\tLoss:\t0.3102306640406862\n",
      "Accuracy:\t0.914875\n",
      "iteration:\t11\t\t\tLoss:\t0.29653488692776153\n",
      "Accuracy:\t0.923\n",
      "iteration:\t12\t\t\tLoss:\t0.2840567491064455\n",
      "Accuracy:\t0.9285\n",
      "iteration:\t13\t\t\tLoss:\t0.2726299003449968\n",
      "Accuracy:\t0.9335\n",
      "iteration:\t14\t\t\tLoss:\t0.2621209529888826\n",
      "Accuracy:\t0.938125\n",
      "iteration:\t15\t\t\tLoss:\t0.25242086733507385\n",
      "Accuracy:\t0.942\n",
      "iteration:\t16\t\t\tLoss:\t0.24343852778200478\n",
      "Accuracy:\t0.945\n",
      "iteration:\t17\t\t\tLoss:\t0.23509641519765603\n",
      "Accuracy:\t0.95\n",
      "iteration:\t18\t\t\tLoss:\t0.22732798212789818\n",
      "Accuracy:\t0.952625\n",
      "iteration:\t19\t\t\tLoss:\t0.22007591629360526\n",
      "Accuracy:\t0.954375\n",
      "iteration:\t20\t\t\tLoss:\t0.21329069699653996\n",
      "Accuracy:\t0.95725\n",
      "iteration:\t21\t\t\tLoss:\t0.20692930236876478\n",
      "Accuracy:\t0.960625\n",
      "iteration:\t22\t\t\tLoss:\t0.20095410510548303\n",
      "Accuracy:\t0.961875\n",
      "iteration:\t23\t\t\tLoss:\t0.19533197418926704\n",
      "Accuracy:\t0.963625\n",
      "iteration:\t24\t\t\tLoss:\t0.19003355914196454\n",
      "Accuracy:\t0.965375\n",
      "iteration:\t25\t\t\tLoss:\t0.18503271973272267\n",
      "Accuracy:\t0.9665\n",
      "iteration:\t26\t\t\tLoss:\t0.1803060671383974\n",
      "Accuracy:\t0.967125\n",
      "iteration:\t27\t\t\tLoss:\t0.17583259005215468\n",
      "Accuracy:\t0.968375\n",
      "iteration:\t28\t\t\tLoss:\t0.17159334616586433\n",
      "Accuracy:\t0.969375\n",
      "iteration:\t29\t\t\tLoss:\t0.16757120470613077\n",
      "Accuracy:\t0.971125\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8XPV97//XZzaNRvtmLbblfTcGg2MCZksDxCEBkjahkNCE2xCaNPSR/VeSX5tQenOb9N6kzf2FX3KB0NBsDtmIQyGENhiHHdlgvGFb3uVF+y6NZvvcP86RPMiSPbaW0Yw+z8djHnPOmTMzn6+O9J6j7/nOOaKqGGOMyS6edBdgjDFm/Fm4G2NMFrJwN8aYLGThbowxWcjC3RhjspCFuzHGZCELd2OMyUIW7ibricghEbk23XUYM5ks3I0xJgtZuJtpS0Q+LiL1ItImIhtFpMZdLiLyLyLSJCKdIvKGiKx0H7tBRHaJSLeIHBORL6S3FcaMzMLdTEsi8ifAPwG3ANXAYWCD+/D1wFXAYqAY+HOg1X3s+8BfqWoBsBL4wySWbUzKfOkuwJg0+TDwsKpuBRCRLwHtIjIXiAIFwFLgFVXdnfS8KLBcRLapajvQPqlVG5Mi23M301UNzt46AKrag7N3PlNV/wB8B7gfaBSRB0Sk0F31z4AbgMMi8qyIXDbJdRuTEgt3M10dB+YMzohIHlAGHANQ1f+tqpcAK3C6Z77oLn9VVW8GZgCPAY9Oct3GpMTC3UwXfhEJDt5wQvm/ichFIpID/A/gZVU9JCJvE5FLRcQP9AJhIC4iARH5sIgUqWoU6ALiaWuRMWdg4W6miyeA/qTblcDfA78ETgALgFvddQuBB3H60w/jdNf8L/exvwAOiUgX8Ang9kmq35hzInaxDmOMyT62526MMVnIwt0YY7KQhbsxxmQhC3djjMlCafuGanl5uc6dOzddb2+MMRlpy5YtLapacbb10hbuc+fOpa6uLl1vb4wxGUlEDp99LeuWMcaYrJSR4R6NJ9JdgjHGTGkZF+7//uIhLv/6HxiI2be+jTFmNCmFu4isF5E97oUN7hnh8VoReUZEXnMvbHDD+JfqqC0N0dw9wLN7mifqLYwxJuOdNdxFxItz6tN3A8uB20Rk+bDV/g54VFVX45yf4/8f70IHrVtYTknIz2/fODFRb2GMMRkvlT33tUC9qh5Q1QjO1WpuHraO4pxsCaAI53SqE8Lv9bB+ZTX/uauRvkhsot7GGGMyWirhPhM4mjTf4C5Ldi9wu4g04Jx972/GpbpR3HhhNf3ROH94s2ki38YYYzJWKuEuIywbfirJ24AfqOosnKvU/FBETnttEblLROpEpK65+fz7zC+dV0ZFQQ6Pb7OuGWOMGUkq4d4AzE6an8Xp3S4fw70ijaq+CASB8uEvpKoPqOoaVV1TUXHWL1iNyusR3nNBNX/Y00R3OHrer2OMMdkqlXB/FVgkIvNEJIBzwHTjsHWOAO8EEJFlOOE+ocNZbrywmkgswdO7GifybYwxJiOdNdxVNQbcDTwF7MYZFbNTRO4TkZvc1T4PfFxEtgE/Be7QCb4KyOrZJdQUBXncRs0YY8xpUjq3jKo+gXOgNHnZV5KmdwHrxre0M/N4hPdeWMPDzx2koy9CcSgwmW9vjDFTWsZ9QzXZjatqiCWU3+04me5SjDFmSsnocF85s5A5ZSHrmjHGmGEyOtxFhBtX1fDC/haauwfSXY4xxkwZGR3uADdeWENC4Xc7bO/dGGMGZXy4L6kqYNGMfH5rX2gyxpghGR/u4Oy9v3KojROd/ekuxRhjpoSsCPf3rqoG4D/swKoxxgBZEu7zK/JZUVNopwE2xhhXVoQ7OF0z2452cKS1L92lGGNM2mVNuL/nAqdr5vHtE3YqeWOMyRhZE+6zS0Osri22UTPGGEMWhTs4pyPYfaKL+qaedJdijDFplVXh/p5V1YjA429Y14wxZnrLqnCvLAyydm4pv912nAk+47AxxkxpWRXu4Iya2d/cy5snu9NdijHGpE3Whfu7V1bh9Qi/3WZdM8aY6Svrwr0sP4fLF5Tx+BsnrGvGGDNtZV24g9M1c6StjzcaOtNdijHGpEVWhvu7llfh91rXjDFm+kop3EVkvYjsEZF6EblnlHVuEZFdIrJTRH4yvmWem6KQn6sXV/Af20+QSFjXjDFm+jlruIuIF7gfeDewHLhNRJYPW2cR8CVgnaquAD4zAbWek/euquFEZ5gtR9rTXYoxxky6VPbc1wL1qnpAVSPABuDmYet8HLhfVdsBVLVpfMs8d9curyTX7+V/PLGb3oFYussxxphJlUq4zwSOJs03uMuSLQYWi8jzIvKSiKwf6YVE5C4RqRORuubm5vOrOEX5OT7+5c8vZNvRDu76YR3haHxC388YY6aSVMJdRlg2vCPbBywCrgFuAx4SkeLTnqT6gKquUdU1FRUV51rrOVu/spr/+YELeb6+lbt/spVoPDHh72mMMVNBKuHeAMxOmp8FDB+G0gD8RlWjqnoQ2IMT9mn3Z5fM4h9vXsF/7m7isz97nbgdYDXGTAOphPurwCIRmSciAeBWYOOwdR4D3gEgIuU43TQHxrPQsfiLy+Zyz7uX8vgbJ/jyr7bbCBpjTNbznW0FVY2JyN3AU4AXeFhVd4rIfUCdqm50H7teRHYBceCLqto6kYWfq09cvYC+gRj/+w/1hHK8fOW9yxEZqcfJGGMy31nDHUBVnwCeGLbsK0nTCnzOvU1Zn71uMT0DcR5+/iD5OT4+f/2SdJdkjDETIqVwzxYiwt+/dxl9kRj/3x/qCQV8fPKaBekuyxhjxt20CndwAv5r77+Avkicb/zuTfJyvHzksrnpLssYY8bVtAt3AK9H+OYtF9IXifOV3+wkFPDxgUtmpbssY4wZN1l54rBU+L0evvOh1axbWMb/84ttPLHdLqxtjMke0zbcAYJ+Lw9+ZA2ra0u4+ydbuXfjTrrD0XSXZYwxYzatwx0gFPDxyF+u5fa3z+GRFw9x7bee5Xc77EIfxpjMNu3DHZzz0Nx380p+/dfrKM3L4RM/2sqdj9TR0N6X7tKMMea8WLgnuWh2Mb+9ex1/955lvLC/leu+tZkHNx8gZuekMcZkGAv3YXxeD3deOZ+nP3cV6xaW8bUndnPjd57nNTsvvDEmg1i4j2JWSYgHP7KG791+Ce29Ef70uy/w94/toMsOuBpjMoCF+xmICOtXVvGfn7+aOy6fy49fPsy133yWn75yhIGYnR/eGDN1WbinID/Hx1dvXMFjn1pHdXEuX/rVdq78xjM8sHm/DZ00xkxJkq4hf2vWrNG6urq0vPdYqCrP17fy3Wfreb6+lcKgj7+4bA53XD6PioKcdJdnjMlyIrJFVdecdT0L9/O37WgH33t2P7/beZKA18Mta2bz8SvnU1sWSndpxpgsZeE+ifY39/Dg5gP8cmsD8YTy3lU1fOLqBSyvKUx3acaYLGPhngYnO8M8/PxBfvzSYXojca5cVM6H1tbyzmWVBHx2eMMYM3YW7mnU2Rflhy8d4kcvHeFkV5iSkJ/3rZ7JBy+ZbXvzxpgxsXCfAuIJ5Y/7mvl5XQNP72okEk+wcmYht6yZzU0X1lAcCqS7RGNMhhnXcBeR9cC3ca6h+pCqfn2U9T4A/Bx4m6qeMbmnQ7gna++N8JvXj/HzLQ3sPN5FwOvh+hWVfHDNbK5YWI7XY9dzNcac3biFu4h4gb3AdUAD8Cpwm6ruGrZeAfAfQAC428J9dDuPd/LzugYee/0YHX1RqouC3HhhDetXVnHRrGI8FvTGmFGkGu6pXIlpLVCvqgfcF94A3AzsGrbePwL/DHzhHGuddlbUFLHipiK+dMNS/mt3Ez+vO8q/PX+QBzYfoKowyLtWVLJ+ZTVr55XaHr0x5rykEu4zgaNJ8w3ApckriMhqYLaqPi4io4a7iNwF3AVQW1t77tVmmRyflxsuqOaGC6rp7I/yhzcbeXL7STa8epRHXjxMWV6A65ZXsn5lFZcvKLcRN8aYlKUS7iPtOg715YiIB/gX4I6zvZCqPgA8AE63TGolTg9FuX7ev3oW7189i75IjE17mnlyx0l+u+04G149SkHQx7XLKnnXiirWLSyjIOhPd8nGmCkslXBvAGYnzc8CjifNFwArgU0iAlAFbBSRm87W725GFgr4hvbow9E4z9e38OSOkzy9q5Ffv3YMn0e4eE4JVy+u4OrFFSyvLrR+emPMW6RyQNWHc0D1ncAxnAOqH1LVnaOsvwn4gh1QHX/ReIK6Q+1s3tfM5r3N7DzeBUB5foArFzlBf8Wicsrz7Rw3xmSrcTugqqoxEbkbeApnKOTDqrpTRO4D6lR149jLNanwez1ctqCMyxaU8bfrl9LUHeaPe1vYvK+ZZ/c28+vXjgFwwcwirlpczroF5ayuLSE34E1z5caYyWZfYsoSiYSy43gnz+5pZvO+ZrYe6SCeUPxeYdWsYi6dV8ql88u4ZE4J+Tmp9MYZY6Yi+4bqNNcVjrLlUDsvHWzl5QNtbD/WSTyheD3CyplFTtjPK2XN3FKKcu3grDGZwsLdvEXvQIytR9p5+UAbLx9sZdvRTiLxBCKwrKqQi+cUs3p2CRfVFjO/PA/34LgxZoqxcDdnFI7Gee1IBy8fbOXVQ21sO9pJz0AMcIZlXjS7mNW1xayuLeGiWcUUhWzv3pipYDy/oWqyUNDvHTo4C85JzvY39/DakXZeO9LB60c7+PZ/7WPws39BRR6ra0tYNauIlTOLWFZVaAdqjZnCbM/djKo7HGV7QyevHe0YCv3W3ggAHoGFM/JZObOIlTVO4C+vKbSDtcZMMNtzN2NWEPRz+cJyLl9YDjjXjz3RGWbHsU52HO9ix7FOntvXwq+2OkMwRWBeeR4ra4pYUVPI0upCllUVUFGQY334xkwyC3eTMhGhpjiXmuJcrl9RNbS8qSvMzuNdbD/WyY5jnWw53M7Gbae+xFwS8rO0qpAlVQUsqy5gSVUhiyvzCQXs18+YiWJ/XWbMZhQGmVEY5B1LZwwt6+iL8ObJbt480cWexm52n+jmZ68epT8aB5y9/DmlIZZUFbCksoAFM/JZNKOA+RV5BP3Wl2/MWFm4mwlRHArw9vllvH1+2dCyREI52t7H7hPd7DnZzZsnu9hzspundzWScA/9iEBtaYiFFfksnPHWm50szZjUWbibSePxCHPK8phTlsf6lae6dcLROIdae6lv6mFfYw/1zT3UN/bwx30tROKJofWqCoPMK89jXkUe88vznOnyPGaXhvB77XTIxiSzcDdpF/R7WVpVyNKqt148PBZPcLS9n32N3U7gN/VwqKWXJ7efoL0vOrSe1yPUloaGwn5eeR5zykLMKc2jpjiIz4LfTEMW7mbK8nk9Q2F9/bDH2nsjHGzt5WBzLwdbnNuBll5e2N9COHpqb9/nEWaW5FJbGhoK/Nqy0NC8HdQ12cp+s01GKskLUJIX4OLakrcsTySUxu4wh1r6ONLWy5G2Pg639nGkrY/fbjtBZ3/0LeuX5+cwqySX2aUhZpXkOtMlznRNca4d3DUZy8LdZBWPR6guyqW6KHfo27fJOvuiHG7rHQr8I619NHT08UZDB09uP0Es8dYv9VUW5jDLDfuZ7jDQmcW5VBcHqSnOpdAO8popysLdTCtFIT+rQsWsmlV82mPxhNLYFaahvZ+G9j6Otjn3De39bDnczn+8cXr4F+T43LH/waHvANQUB6kqzKWqKEhVYdBO02DSwsLdGJfXc+pLWmvnlZ72eDyhtPQMcKyjn+NDt7Bz39nPtoZO2tzTMyQrDvmpKgxSVRSkusgJ/uqiIJVu+M8oyKE45Ldv8ZpxZeFuTIq8HqGyMEhlYfC0vv5B/ZE4xzv7aewMc7IrzInOMCfd6ZOdYXYc66KlZ+C05wV8HmYU5FDphn1lYZAZhTlUFjj3MwrsQ8CcGwt3Y8ZRbsDLgop8FlTkj7pOJJagqdsJ+8auARq7wjR2h2lyp/c19fBcfQvd4dhpz/V7hfL8HCoKcqhw72cUuPPurTw/h7L8HPICXvsgmMZSCncRWQ98G+caqg+p6teHPf454E4gBjQDf6mqh8e5VmOyQsDncQ/Shs64Xl8kNhT4jd0DtHQP0NQ9QHP3AM09AxzvDLOtoZPW3gFGOrlr0O+hLC+H8oIcKvID7nRgaFl5XoBSd3lJyG/fB8gyZw13EfEC9wPXAQ3AqyKyUVV3Ja32GrBGVftE5JPAPwN/PhEFGzNdhAI+5pb7mFued8b1YvEEbX0Rmt3wb+2J0NIzQGvPAC3u9LGO8NAxgXhi5NN8F4f8lOYFKMsLUJoXoDQvh/L8wekAJSH3Pi9AaShgB4qnuFT23NcC9ap6AEBENgA3A0PhrqrPJK3/EnD7eBZpjBmdz+tx++SDrDjLuomE0tEfpbXH2ftv643Q1huhtScyNN3SM8CB5l7qDrXT3hdhlM8Ccnye00K/JOSnOBSgONdPSV7SdMhZryDow+OxrqLJkEq4zwSOJs03AJeeYf2PAU+O9ICI3AXcBVBbW5tiicaY8eLxyNCe+KLKgrOuH08onf1R2nojtPdFaHfv23qj7n2EDvf+WEc/bb0RusLREbuJwLnIS5Eb9kUhP0W5fopznQ+BwqFpd7l7X5QboDDXR47P/lM4F6mE+0gfsyNuOhG5HVgDXD3S46r6APAAOFdiSrFGY0yaeJM+DFIVTyhd/U74d/RH6eiL0N4bPTXdF6G9L0pXf5TWnggHmnvp6IvQNcIB5GRBv4fC4GDg+ykcvA/6huYL3fnC4OC0n4Kgj4Kgb9odU0gl3BuA2Unzs4Djw1cSkWuB/xe4WlVPH+tljJkWvB4ZOj3EuYgnlO5wlI6+KJ39pz4Muvqd+a5wjM6+wekojV1h9jZ209UfpXsgNup/C4PyAl4Kgn4Kc30UDIX+qfAf/CDIzzl9eX6Oj/ygL6POPppKuL8KLBKRecAx4FbgQ8kriMhq4P8A61W1adyrNMZkPa9HnD760Ll9KIDzwdATjtEVdoK/qz9Gd9j5QOhyPwy6k6a7+mO09kQ41NJLdzhGdzj2ltNLjybH5xn6AMgfvM9x/lvIy3FuBUEfeQEv+UE/+Tle8nIG13Mezw/6yAv48E7wsYezhruqxkTkbuApnKGQD6vqThG5D6hT1Y3A/wTygZ+742qPqOpNE1i3McYM8XrE6cMPnf+5fsLRuBv0UXoGYkPT3eEYPQMxetz77uTpcJSG9j7ncXf58FNUjOS+m1fwkcvmnnetqUhpnLuqPgE8MWzZV5Kmrx3nuowxZlIF/V6Cfi8VBTnn/RqqykAsQe9AjN6BON0DUXoH4vS6Hwq97m3NnNNPbzHe7BuqxhgzTkRk6EOibPQvKU+KzDk6YIwxJmUW7sYYk4VEzzZ+aKLeWKQZON/zz5QDLeNYzlSQbW3KtvZA9rUp29oD2demkdozR1UrzvbEtIX7WIhInaquSXcd4ynb2pRt7YHsa1O2tQeyr01jaY91yxhjTBaycDfGmCyUqeH+QLoLmADZ1qZsaw9kX5uyrT2QfW067/ZkZJ+7McaYM8vUPXczjYnIJhFpF5Hz/yqhMVnOwt1kFBGZC1yJc9rpSTt/kYjYt7lNRsm4cBeR9SKyR0TqReSedNczViJySES2i8jrIlKX7nrOh4g8LCJNIrIjaVmpiDwtIvvc+5JxeruP4Fzt6wfAR5PeL1dEvikih0WkU0SeE5Fc97ErROQFEekQkaMicoe7fJOI3Jn0Gne4zxtsj4rIp0RkH9AsIsdEpFlEIiLSJyJbROTKpOd7ReTLIrJfRLrdx2eLyP0i8s1hP7PfishnxulnclZuHc+IyG4R2Skin3aXT9R2mlBnaM+97nZ63b3dkO5aUyUiQRF5RUS2uW36B3f5PBF52d1GPxOR1E6bqaoZc8M5K+V+YD4QALYBy9Nd1xjbdAgoT3cdY2zDVcDFwI6kZf8M3ONO3wN8Y5zeqx74a+ASIApUusvvBzbhXDnMC1wO5AC1QDdwG+AHyoCL3OdsAu5Meu07gOeS2qPA00Ap8N+BL+BcQrIM57xMnwdOAkH3+V8EtgNLcC5yc6G77lqcayB43PXKgb7B2idpG1UDF7vTBcBeYPlEbac0tude4Avpru882yRAvjvtB14G3g48CtzqLv8e8MlUXi/T9tyHrueqqhFg8HquJo1UdTPQNmzxzcAj7vQjwPvG+j4icgUwB3hUVbfgfNB/SEQ8wF8Cn1bVY6oaV9UX1LlozIeB/1TVn6pqVFVbVfX1c2jPP6lqGxBzH/uR+xoxVf0mzgfIEnfdO4G/U9U96tjmrvsK0Am8013vVmCTqjaO9WeSKlU9oapb3eluYDfOB+G4b6fJcIb2ZCz3d6bHnfW7NwX+BPiFuzzlbZRp4T7S9VwzeoPibLzfu//C35XuYsZRpaqeAOcPEZgxDq/5UeD3qjr4deyfuMvKgSBO2A83e5TlqUr+fbtbRI673TudItIBFLnvf7b3eoRTF46/HfjhGGoaE/e4xWqcPcOJ2E6Talh7wNlOb7jdaxnRzTTI7dp7HWjC+a9xP9ChqoPXIEw58zIt3FO+nmsGWaeqFwPvBj4lIlelu6CpyO0/vwW4WkROishJ4LM4XR/VQBhYMMJTj46yHKAXCCXNV42wzuDv13dxPkh8OP8m/1JVi3H2yAd/L8/0Xj8CbhaRC4FlwGOjrDehRCQf+CXwGVXtSkcN42mE9nwXZxtcBJwAvnmGp0857n+dF+FcznQtzu/Kaaul8lqZFu4pXc81k6jqcfe+Cfg1zgbNBo0iUg3g3o/18ovvA+I4/aoXubdlwB9xDrI+DHxLRGrcvZ/L3KGSPwauFZFbRMQnImUicpH7mq8DfyoiIRFZCHxstDd3u1DycLpnHgQuFZGvAIVJqz0E/KOILBLHKhEpc5/fgHPJyh/ifDD0j/Hncc5ExI8ThD9W1V+5i8d7O02akdqjqo1uQCZwtlNG/j2pagfOMaG3A8VyarRWypmXaeE+dD1X94jxrcDGNNd03kQkT0QKBqeB64EdZ35WxtjIqdEsHwV+M8bX+yjwb6p6RFVPDt6A7+D0q9+DczDzVZz+8m/gHMA8AtyAc/CzDSfQL3Rf81+ACNCI023y49He3A2+p4AncT5Q5uP8t5DcbfMtnL363wNdwPeB3KTHHwEuIA1dMiIibj27VfVbSQ+N93aaFKO1Z/CDyvV+MujvSUQqRKTYnc4FrsU5lvAM8AF3tZS3UcZ9Q9Ud2vSvnLqe69fSXNJ5E5H5OHvr4Py7/5NMbI+I/BS4BqfvuRH4Kk63w6M4o1WOAB90D0xOeaO05xqc/xYUZ4TTXw32VZ/D616F0z0z192znDTuweg/4nwADr73l3H6qTNuO52hPbcxxu2ULiKyCmcHwIuz4/2oqt7n5sQGnFFbrwG3u4MFzvx6mRbuxmQitwthA7BNVe9Ldz0m+2Vat4wxGUdElgEdOAd+/zXN5ZhpwvbcjTEmC9meuzHGZKG0nQypvLxc586dm663N8aYjLRly5YWTeEaqmkL97lz51JXl5HnyTLGmLQRkcOprGfdMsYYk4XsHNXGGDOOEgklHIsTjiYIR+PuLUF/NM5ANE44FmdxZQGzSkJnf7ExsHA3xkxJqspALMFANEE0kSAaTxCNKZG4O+3eIjEdmo4llERCiasSTwy7qfNYzJ0fiA0+/9R9xH29SDxB1J0feu24Ek0osbfMu/fu88Mx5/5s/vv7VnL72+dM6M/Pwt0Yc0aRWIKBWNwNQXVDMH4qBJOCcSDqrBuOxumPxAnHEkN7rqf2Yk/tyYajzjoDScvDMee5AymE5HjweYSAz4Pf6yHg8xBw7/1ewe91l7vLQl4Pfo+z3Oc+7vMIfvd5OX4PQZ+XoN9Lrt9D0O91bx5y/F6CPi+5AS+zS3LPXthY2zXh72CMSavBPeCucJTucMy9Renqj9HRH6GzP0pnX5SOviid/VE6+iN09EXp6o/S0R+lLxIfcw2DwZebFHZBN+yKcv0EC3LeutzvJejzEAx4nee64ev3etwgPRW8zk3wuUHr9Qg+j+DxCF5x5t9yE8HrFSewvR48npFONpv5LNyNmULiCaU3EqMnHKNnwAni3gHnNtiP6+wRO9MD0fipPWB377g7HKN74K1BHo2f+cuKAZ+HkpCf4twARbl+ZpeGKM71UxzyUxj0kxvwntq7HQpYDwGfEPB68Xudvd+A71Q45w7usfq8eLM0QKcyC3djJlA0nqC5e4DGrjCNXYP3znRTd5iu/ijdA6fC/Fz3knPcMM1N2uvNz/FRkZ/D/PJ8CoI+CoJ+CoI+CoM+CnP9b1lWEnLCPOj3TtBPwKSLhbsxKYjFE0N7wl1hp8uiKxylKxxzp5377nCM9r7IUIC39g4w/AwfPo8woyCHisIgxaEAs0pDFOT4yM/xkR907guCPvJyTk2HAr7TujRyfB6cM98aczoLdzMthaNxmrsHaOkZoL0vQltvlLbeAdp6o7T3RmjtjbjLnVtnf/SMrycC+Tk+CoNOV0ZlYZBVs4qpLMyhsjBIZWEOMwqCVBUFKQ0Fsraf10wdFu4ma6gqnf3RoS6P5u4BmroHaE66DS7vCsdGfA2/VyjNC1ASClCWH2BFTeHQ/GD/c4HbvVEY9FOY60znB3wW2GZKsXA3GaE/EudEZ/9QcJ/sdPuwu8M0dYU56XaDjDTGOBTwOt0gBTksqSrgioXlzCgMUpGfQ3lBgNK8HEpDAUrzA+QFvNbVYbKChbtJu4FYnJOdYY53hDnR2c+JTve+I8xxd7qj7/RukVDAS1VhkMrCIJfUllBZGGRGUhfIYKDn5divuZl+7LfeTJqOvgh7G3vY19TNvqT7pu7TrxhWEvJTXZTLzOIgl8wpprool+qioBPmRU6g51toGzMq++sw4669N8K+plPhvbexm31NPTQnhXhewMvCygKuWlxBbWmI6qIgNcVOgFcX5ZIbsKF5xoyFhbs5L6pKY9cA9U091Dc54V3v3lp7I0PrDYb41YsrWFyZz6LKAhbNyGdmca71bRszgSzczRl1h6McaevjSGsfh9v62N/Uw76mHvY39dA9cGrESWHQx6L4F2AOAAAN60lEQVTKAq5dVsnCGfksnJHPokoLcWPSxcJ9mlNVmroHONza54Z4L4fb+obm25L2wgEqCnJYWJHP+y+e6YR4RT4LK/OpyM+xEDdmCrFwn2Yau8K80dDJGw0dvNHQyfZjnW8JcI9ATXEuc8pCvGtFFbWlIeaUhagtDVFbFqIw6E9j9caYVFm4Z7G23shQiDtB3kFjl3NQ0+sRFs3I57pllSyvKWRueR5zSkPUFOcS8NkFuozJdBbuWaSpK8yLB1p5ob6VFw+0cqStD3C+Gj+/PI/LF5SzalYRq2YVsby6yEakGJPFLNwzWFtvhJcOtPLi/lZe2N/C/uZewDm4een8Mj58aS2rZhWzcmYhBdadYsy0YuGeQbrDUV4+0MYL+509890nugBnuOHb5pXy52+bzeULyllWXWjnzzZmmrNwn8ISCWXH8U42721m894Wth5pJ5ZQcnwe1swt4QvXL+Yyt6vF77V+cmPMKRbuU0xTV5jN+1rYvLeZ5+pbhkayrKgp5ONXzefKReVcXFtiF1cwxpyRhXuaJRLKK4faeObNJp7d28ybJ7sBKM/P4ZrFFVy1uIIrFpVTnp+T5kqNMZnEwj1NjrT28YstR/nl1mMc6+jH7xXWzCnlb9cv5arF5SyrKrTzgxtjzpuF+yTqHYjxxPYT/GJLAy8fbEMErlhYzt++eynvXDrDTk1rjBk3liYTTFV55WAbP9/SwBPbT9AXiTOvPI8vvmsJf3rxTKqLctNdojEmC1m4T5Dm7gF++soRfrGlgSNtfeTn+Ljpwho+cMksLplTYudhMcZMKAv3cdbRF+F7zx7gkRcO0R+Nc/mCMj5z7SLWr6wiFLAftzFmcljajJPucJSHnzvEQ388QE8kxk0X1vDpdy5ifkV+ukszxkxDFu5j1B+J8+8vHuJ7z+6nvS/Ku1ZU8tnrFrO0qjDdpRljpjEL9/M0EIuz4ZWjfOeZepq7B7hqcQVfuH4xq2YVp7s0Y4yxcD9XsXiCX209xrf/ax/HOvpZO6+U+z90MWvnlaa7NGOMGWLhfg4OtfRy1w/r2NvYw4Wzivj6n13AFQvLbeSLMWbKsXBP0UsHWvnEj7YgwPduv4R3rai0UDfGTFkpnUpQRNaLyB4RqReRe0Z4vFZEnhGR10TkDRG5YfxLTZ9H647yF99/mbK8AI99ah3rV1ZZsBtjprSz7rmLiBe4H7gOaABeFZGNqrorabW/Ax5V1e+KyHLgCWDuBNQ7qRIJ5RtPvcn/efYAVy4q5zsfupiiXLvohTFm6kulW2YtUK+qBwBEZANwM5Ac7goMjv0rAo6PZ5Hp0BeJ8ZkNr/P7XY3c/vZa7r1xBT47Z7oxJkOkEu4zgaNJ8w3ApcPWuRf4vYj8DZAHXDvSC4nIXcBdALW1teda66Q50dnPnY/UsftEF/feuJyPXj7XumGMMRkllV3RkVJNh83fBvxAVWcBNwA/FJHTXltVH1DVNaq6pqKi4tyrnQRvNHRw83ee53BrH9+/423csW6eBbsxJuOksufeAMxOmp/F6d0uHwPWA6jqiyISBMqBpvEocrI8uf0En330dcrycvjlJy9lSVVBuksyxpjzksqe+6vAIhGZJyIB4FZg47B1jgDvBBCRZUAQaB7PQieSqnL/M/V88sdbWV5dyG/uXmfBbozJaGfdc1fVmIjcDTwFeIGHVXWniNwH1KnqRuDzwIMi8lmcLps7VHV4182U9U9PvskDmw9w80U1fOPPVtn1SY0xGS+lLzGp6hM4wxuTl30laXoXsG58S5scP3j+IA9sPsBHLpvDP9y0wvrXjTFZYVqP7fvdjpP8w+O7uH55JV+90YLdGJM9pm24bznczqc3vMZFs4v59q2r8drFqI0xWWRahvvBll7ufORVqoqCPPSRNeQGrI/dGJNdpl24t/YMcMe/vYKI8IP/tpay/Jx0l2SMMeNuWoV7fyTOxx6p42RnmIc+uoZ55XnpLskYYybEtDnlbzyhfHrDa2xr6OC7H76Ei2tL0l2SMcZMmGmx566q/OPju/j9rka++t7lrF9Zle6SjDFmQk2LcH/ojwf5wQuHuPOKedyxbl66yzHGmAmX9eH++BvH+doTu3nPBdV8+YZl6S7HGGMmRVaH+ysH2/jcz7axZk4J37zlQjw2lt0YM01kbbhH4wn++sdbmVWay4MfWWPnizHGTCtZG+5bD7fT0jPAF65fQkleIN3lGGPMpMracN+0txmfR7hiUXm6SzHGmEmXveG+p5mL55RQGLQLWhtjpp+sDPeTnWF2n+jiHUtmpLsUY4xJi6wM92f3Olf3u2bJ1LxOqzHGTLSsDPdNe5qpKgyy1C6VZ4yZprIu3KPxBM/ta+HqxRV28Q1jzLSVdeG+5XA73QMx3rHUumSMMdNX1oX7pj3OEMh1C20IpDFm+srCcG/ikjklFNgQSGPMNJZV4X6yM8ybJ7t5x1IbAmmMmd6yKtw37bEhkMYYA1kX7s4QyCWVNgTSGDO9ZU24R+MJnq9v4R1LbQikMcZkTbgPDoG8erH1txtjTNaE+zN7mtwhkGXpLsUYY9Iua8L92T3NrJlrQyCNMQayJNxPdPY7QyDtLJDGGANkSbhv2tMMwDUW7sYYA2RNuDdRXRRkcWV+uksxxpgpIePDPRJL8Hx9K9csmWFDII0xxpXx4b7lcDs9AzH7VqoxxiTJ+HDftKcJv9fOAmmMMclSCncRWS8ie0SkXkTuGWWdW0Rkl4jsFJGfjG+Zo9u0p5k1c0rJz/FN1lsaY8yUd9ZwFxEvcD/wbmA5cJuILB+2ziLgS8A6VV0BfGYCaj3N8Y5+9jR224U5jDFmmFT23NcC9ap6QFUjwAbg5mHrfBy4X1XbAVS1aXzLHNmze20IpDHGjCSVcJ8JHE2ab3CXJVsMLBaR50XkJRFZP9ILichdIlInInXNzc3nV3GSZ95soqYoyKIZNgTSGGOSpRLuI40v1GHzPmARcA1wG/CQiBSf9iTVB1R1jaquqagYW1eKMwSyhWuW2hBIY4wZLpVwbwBmJ83PAo6PsM5vVDWqqgeBPThhP2HqDrfRG4lzzWLrbzfGmOFSCfdXgUUiMk9EAsCtwMZh6zwGvANARMpxumkOjGehw23a04zfK1xuQyCNMeY0Zw13VY0BdwNPAbuBR1V1p4jcJyI3uas9BbSKyC7gGeCLqto6UUWDM7597TwbAmmMMSNJKRlV9QngiWHLvpI0rcDn3NuEO97Rz97GHj54yeyzr2yMMdNQRn5D9dRZIK2/3RhjRpKR4f7MniZmFuey0IZAGmPMiDIu3COxBC/Ut3DNErsQtjHGjCbjwr3ukDsE0r6Vaowxo8q4cN9yuJ2A18PlC+xC2MYYM5qMG0f4N+9cxAfXzCbPhkAaY8yoMm7PHaCqKJjuEowxZkrLyHA3xhhzZhbuxhiThcT5cmka3likGTh8nk8vB1rGsZypINvalG3tgexrU7a1B7KvTSO1Z46qnvUbnGkL97EQkTpVXZPuOsZTtrUp29oD2dembGsPZF+bxtIe65YxxpgsZOFujDFZKFPD/YF0FzABsq1N2dYeyL42ZVt7IPvadN7tycg+d2OMMWeWqXvuxhhjzsDC3RhjslDGhbuIrBeRPSJSLyL3pLuesRKRQyKyXUReF5G6dNdzPkTkYRFpEpEdSctKReRpEdnn3peks8ZzMUp77hWRY+52el1EbkhnjedKRGaLyDMisltEdorIp93lGbmdztCejN1OIhIUkVdEZJvbpn9wl88TkZfdbfQz91rWZ3+9TOpzFxEvsBe4DmjAuXj3baq6K62FjYGIHALWqGrGfvFCRK4CeoB/V9WV7rJ/BtpU9evuh3CJqv5tOutM1SjtuRfoUdX/lc7azpeIVAPVqrpVRAqALcD7gDvIwO10hvbcQoZuJ3EuUJGnqj0i4geeAz6Nc/nSX6nqBhH5HrBNVb97ttfLtD33tUC9qh5Q1QiwAbg5zTVNe6q6GWgbtvhm4BF3+hGcP7yMMEp7MpqqnlDVre50N87F7meSodvpDO3JWOrocWf97k2BPwF+4S5PeRtlWrjPBI4mzTeQ4RsUZ+P9XkS2iMhd6S5mHFWq6glw/hCBbLi6yt0i8obbbZMR3RcjEZG5wGrgZbJgOw1rD2TwdhIRr4i8DjQBTwP7gQ5VjbmrpJx5mRbuI11XL3P6lUa2TlUvBt4NfMrtEjBTz3eBBcBFwAngm+kt5/yISD7wS+AzqtqV7nrGaoT2ZPR2UtW4ql4EzMLpqVg20mqpvFamhXsDMDtpfhZwPE21jAtVPe7eNwG/xtmg2aDR7Rcd7B9tSnM9Y6Kqje4fXgJ4kAzcTm4/7i+BH6vqr9zFGbudRmpPNmwnAFXtADYBbweKRWTw6kQpZ16mhfurwCL36HEAuBXYmOaazpuI5LkHgxCRPOB6YMeZn5UxNgIfdac/CvwmjbWM2WAAut5Phm0n92Dd94HdqvqtpIcycjuN1p5M3k4iUiEixe50LnAtzrGEZ4APuKulvI0yarQMgDu06V8BL/Cwqn4tzSWdNxGZj7O3Ds4lD3+Sie0RkZ8C1+CcnrQR+CrwGPAoUAscAT6oqhlxkHKU9lyD86++AoeAvxrsq84EInIF8EdgO5BwF38Zp58647bTGdpzGxm6nURkFc4BUy/OjvejqnqfmxMbgFLgNeB2VR046+tlWrgbY4w5u0zrljHGGJMCC3djjMlCFu7GGJOFLNyNMSYLWbgbY0wWsnA3xpgsZOFujDFZ6P8CZWkHgQqqoo0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time for GD is:\t433.20170760154724\n"
     ]
    }
   ],
   "source": [
    "# GD 算法\n",
    "import time\n",
    "input_dim = 784\n",
    "num_classes = 2\n",
    "mini_batch = 8000\n",
    "model1 = LogisticRegression(input_dim, num_classes, lr=1000, batch=mini_batch, weight_scale=4e-2, weight_decay=1.0, reg=0.0)\n",
    "\n",
    "tstart = time.time()\n",
    "model1.train(train_data[0:mini_batch], train_labels[0][0:mini_batch])\n",
    "tend = time.time()\n",
    "\n",
    "# 画图\n",
    "Loss, Acc = model1.history()\n",
    "plt.figure\n",
    "plt.subplot(2,1,1)\n",
    "plt.plot(Loss)\n",
    "plt.title('Loss')\n",
    "plt.subplot(2,1,2)\n",
    "plt.plot(Acc)\n",
    "plt.title('Accuracy')\n",
    "plt.show()\n",
    "print('Time for GD is:\\t%s' % (tend - tstart))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用 SGD 进行过拟合\n",
    "使用 SGD 算法和上面的进行比较，为了公平起见仍然选取 8000 的数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iteration:\t0\t\t\tLoss:\t1.0713048845677386\n",
      "Accuracy:\t0.385\n",
      "iteration:\t1\t\t\tLoss:\t4.296362881919019\n",
      "Accuracy:\t0.723\n",
      "iteration:\t2\t\t\tLoss:\t1.375619632620113\n",
      "Accuracy:\t0.733\n",
      "iteration:\t3\t\t\tLoss:\t1.3455019188032393\n",
      "Accuracy:\t0.834\n",
      "iteration:\t4\t\t\tLoss:\t0.8196256356601929\n",
      "Accuracy:\t0.74\n",
      "iteration:\t5\t\t\tLoss:\t1.460220411803774\n",
      "Accuracy:\t0.788\n",
      "iteration:\t6\t\t\tLoss:\t0.7302072829584296\n",
      "Accuracy:\t0.787\n",
      "iteration:\t7\t\t\tLoss:\t0.8944914994914079\n",
      "Accuracy:\t0.842\n",
      "iteration:\t8\t\t\tLoss:\t0.34538405726515153\n",
      "Accuracy:\t0.867\n",
      "iteration:\t9\t\t\tLoss:\t0.44174882930282583\n",
      "Accuracy:\t0.88\n",
      "iteration:\t10\t\t\tLoss:\t0.2523938682443823\n",
      "Accuracy:\t0.914\n",
      "iteration:\t11\t\t\tLoss:\t0.15520550960735904\n",
      "Accuracy:\t0.963\n",
      "iteration:\t12\t\t\tLoss:\t0.08124154100562958\n",
      "Accuracy:\t0.98\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VPW9//HXJ8tk3xcgCSEJ+6YoKCCg4opLxavetkpba7Xqr/XeLt62ett6u9zeW2839dbbW2uttlq9KlitBTfAIipKCMoOCUsg+75vk+Tz+2MmGCBASGYymZnP8/HIY2bOnDnnc1je+c73fM/3iKpijDEmsIT4ugBjjDGeZ+FujDEByMLdGGMCkIW7McYEIAt3Y4wJQBbuxhgTgCzcjTEmAFm4m4AnIodE5DJf12HMSLJwN8aYAGThboKWiHxZRIpEpE5EXhGRDPdyEZFfiUiViDSKyDYRmeV+72oR2SUizSJSKiL/4tujMGZgFu4mKInIJcB/Ap8GxgHFwHPut68ALgSmAInAZ4Ba93u/B+5S1ThgFrBuBMs2ZtDCfF2AMT6yAnhCVQsAROR+oF5EcgAnEAdMAz5U1d39PucEZojIx6paD9SPaNXGDJK13E2wysDVWgdAVVtwtc4zVXUd8GvgUaBSRB4TkXj3qjcCVwPFIvJ3EVk4wnUbMygW7iZYlQET+l6ISAyQApQCqOojqjoXmImre+Zb7uWbVXU5kA78BXh+hOs2ZlAs3E2wCBeRyL4fXKF8m4jMEZEI4D+AD1T1kIicJyLzRSQcaAU6gB4RcYjIChFJUFUn0AT0+OyIjDkFC3cTLFYD7f1+lgDfB1YC5cBE4LPudeOB3+HqTy/G1V3zc/d7nwcOiUgTcDfwuRGq35gzInazDmOMCTzWcjfGmABk4W6MMQHIwt0YYwKQhbsxxgQgn12hmpqaqjk5Ob7avTHG+KUtW7bUqGra6dbzWbjn5OSQn5/vq90bY4xfEpHi069l3TLGGBOQLNyNMSYAWbgPoLnDyY2/eY/399eefmVjjBmFLNwHsGZ7BVuK63l0fZGvSzHGmCGxcB/AiwUlAGwsquFgTauPqzHGmDNn4X6cI3VtfHiwjlsXTiAsRHj2w8O+LskYY86YR8NdREJFZKuIvOrJ7Y6kVQWlANx50USumDmGF/KP0OG0WV2NMf7F0y33rwG7T7vWKKWqrNpawsK8FDITo1gxfwL1bU7W7Cj3dWnGGHNGPBbuIpIFXAM87qltjrQtxfUU17Zx49wsABbmpZCbGsMzm6xrxhjjXzzZcn8I+DbQe7IVROROEckXkfzq6moP7tozVhaUEhUeyrJZYwEICRFuOT+b/OJ69lQ0+bg6Y4wZPI+Eu4hcC1Sp6pZTraeqj6nqPFWdl5Z22qkRRlSHs4dXt5Vx1ayxxEZ8MivDTXOzcISFWOvdGONXPNVyXwRcJyKHgOeAS0TkaQ9te0S8tbuS5o5ubjg365jlSTEOrp09jpe2ltLa2e2j6owx5sx4JNxV9X5VzVLVHFz3oVynqn51b8lVBaWMS4hk4cSUE95bsSCbls5uXvm4zAeVGWPMmbNx7kB1cyd/31fN9edkEhoiJ7x/bnYS08bG8fSmYuyes8YYf+DxcFfVt1X1Wk9v15te/qiUnl7lxnMzB3xfRFixYAI7y5r4uKRxhKszxpgzZy13XKNkzs5KYFJ63EnXuX5OBtGOUJ7ZNKiplI0xxqeCPtx3lTWxu7zphBOpx4uLDGf5nEz+uq2MxjbnCFVnjDFDE/ThvqqghPBQ4VNnZ5x23RXzs+lw9rLSPbGYMcaMVkEd7t09vfzlozKWTk0nOcZx2vVnZSYwZ3wiz3xgJ1aNMaNbUIf7O4U11LR0Hp1uYDBWzM9mf3UrHxys82JlxhgzPEEd7isLSkiKDmfp1PRBf+ZTZ2cQHxnG03Zi1RgzigVtuDe2O3ljVyXXnZ2BI2zwfwyR4aHcNHc8r++soLq504sVGmPM0AVtuK/eXk5Xd+9pR8kM5Jb52Th7lBe2HPFCZcYYM3xBG+6rCkqYlB7LWVkJZ/zZSemxLMhL5s8fHKan106sGmNGn6AM9+LaVjYfqueGczMROXG6gcH43IIJlNS3s6Fw9E1dbIwxQRnuqwpKEYF/OGfg6QYG44oZY0mNddhUwMaYUSnowr2313UrvUUTUxmXEDXk7TjCQvj0vPGs21NJWUO7Bys0xpjhC7pwzy+u50hdOzecZJKwM3Hz+dko8NyH1no3xowuQRfuK7eUEO345FZ6wzE+OZqLp6Tx3OYjOHtOendBY4wZcUEV7h3OHv62vZyrZo0j2hF2+g8Mwor5E6hq7mTt7kqPbM8YYzwhqML99Z0VtHR2c+Pc4XfJ9Fk6LZ2MhEiethOrxphRJKjCfVVBKZmJUSzIPfFWekMVGiLcfH42G4tqOFjT6rHtGmPMcARNuFc2dfBOYTX/cE4mIQPcSm84PnPeeEJDhGftxKoxZpQImnB/+aNSehX+wQOjZI6XHh/JFTPG8EL+ETqcPR7fvjHGnKmgCHdVZeWWUuaMT2RiWqxX9vG5BROob3OyZke5V7ZvjDFnIijCfWdZE3srm89o3vYztTAvhdzUGLti1RgzKgRFuK8qKMURGsKnzhrntX2EhAi3nJ9NfnE9eyqavLYfY4wZjIAPd2dPLy9/VMql09NJjD79rfSG46a5WTjCQqz1bozxuYAP9w37qqlt7RrSvO1nKinGwbWzx/HS1lJaO7u9vj9jjDmZgA/3lQUlJMc4uHhq2ojsb8WCbFo6u3nl47IR2Z8xxgwkoMO9sc3JW7uquO7sDMJDR+ZQz81OYtrYOJ7eVIyq3cjDGOMbAR3uf91WRldPLzd5cZTM8USEFQsmsLOsiY9LGkdsv8YY019Ah/uqghKmjIllZkb8iO73+jkZRDtCeWZT8Yju1xhj+gRsuB+obqHgcAM3nJs15FvpDVVcZDjL52Ty121lNLY5R3TfxhgDARzuL20tJWSYt9IbjhXzs+lw9rKyoMQn+zfGBLeADPfeXmVVQSmLJqUyJj7SJzXMykxgzvhEnvnATqwaY0ZeQIb7BwfrKG1oH9ETqQNZMT+b/dWtfHCwzqd1GGOCT0CG+6qCEmIjwrhixvBvpTcc156VQXxkGE/biVVjzAgLuHBv6+pm9fZyrp49lihHqE9riXKEctPc8by+s4Lq5k6f1mKMCS4eCXcRGS8i60Vkt4jsFJGveWK7Q/HGzkpau3pGZLqBwbhlfjbOHuWFLUd8XYoxJoh4quXeDdyrqtOBBcBXRWSGh7Z9RlYWlJCVFMX5Ocm+2P0JJqXHsiAvmT9/cJieXjuxaowZGR4Jd1UtV9UC9/NmYDcw4mMQKxo72FhUww1euJXecKyYP4GS+nY2FFb7uhRjTJDweJ+7iOQA5wAfDPDenSKSLyL51dWeD7qXtpaiCv8wSrpk+lw5cyypsQ6bCtgYM2I8Gu4iEgusBL6uqifcsUJVH1PVeao6Ly3Ns7M0qiqrCkqYOyGJ3NQYj257uBxhIXx63njW7amkrKHd1+UYY4KAx8JdRMJxBfszqrrKU9sdrO2ljRRWtXCDF26A7Qk3n5+NAs99aK13Y4z3eWq0jAC/B3ar6i89sc0ztaqgFEdYCNfOzvDF7k9rfHI0F09J47nNR3D29Pq6HGNMgPNUy30R8HngEhH5yP1ztYe2fVpd3a5b6V0+fQwJ0eEjtdsztmL+BKqaO1m7u9LXpRhjAlyYJzaiqhsBnw1PeXtvFfVtTm6cOzq7ZPosnZZORkIkT286zLJZ3rtZtzHGBMQVqqsKSkmNdbBk8sjcSm+oQkOEz56fzcaiGg7WtPq6HGNMAPP7cK9v7WLtnkqWz8kcsVvpDcdnzxtPaIjwrJ1YNcZ40ehPw9N4dVsZzh4dtaNkjpceH8kVM8bwQv4ROpw9vi7HGBOg/D7cXywoZdrYOGaMG9lb6Q3HivkTqG9zsmZHua9LMcYEKL8O96KqFj4+0sCNPriV3nBcMDGF3NQYu2L1DG0raeDdohq7+Ykxg+CR0TK+sqqghBCB5XNG59j2kwkJEW45P5ufrN7Nnoompo31n28dvlBU1cLPXt/D6ztdQ0iXTE7lx8tnkTPKrkQ2ZjTx23Dv7VVe2lrKhVPSSPfRrfSG48a5Wfzsjb18+8VtzMyIJzw0hPDQEBxhrseIsBDCQwVHaAjhYSE43O85jlvP0e+98FAZcJ3wUPGrbzZ9Kho7eHjtPv5v8xGiwkP55uVTiI0I45dv7uOKhzbwlYsncvdFE4kM9+28/caMRn4b7u8fqKW8sYP7r57u61KGJDnGwd0X5rGyoJS1u6vo6unF2d3reuzxbLdDjCOUG+dmcdui3FE3785AGtud/O/f9/PExoP0qvKFhTncc8kkUmMjALjmrHH8+NVdPPRWIS9/VMaPls8c9cNgjRlp4qv+y3nz5ml+fv6QP//N5z/izZ2VbP7eZQHXclPVoyHf1d2Ls6eXLnfw97129vTS2X3qdfoe91e38uq2Mrp7lUunjeH2xbksyEseda35DmcPf3q/mF+vL6Kx3cn1czL45uVTyU6JHnD9DfuqeeDlHRyqbePas8bx/Wtn+OyG6MaMFBHZoqrzTrueP4Z7a2c35/3kLa47O4Of3niWhysLTFVNHfxpUzHPfHCYutYuZoyL5/bFuXzq7AwcYb49r97j7mL75Rt7KWvs4MIpaXz7yqnMykw47Wc7nD3879/38z9v78cRGsK9V0zhCwtzCB1F8/kb40kBHe4rt5Rw7wsf88LdCzlvlNxxyV90OHt4aWspT2w8SGFVC2lxEXxhwQRWLJhAcoxjRGtRVdbtqeK/XtvL3spmzspK4L5l07hgUuoZb+tgTSsPvLyDdwprmJUZz0+un83Z4xO9ULUxvhXQ4X7L7zZxpL6NDd9aOuq6FvyFqrKhsIbfbzzIhn3VRISFcMO5Wdy+OIdJ6XFe3/+W4noeXLOHDw/VkZMSzbeunMbVs8cO6+9TVfnb9nJ+9NddVLd0smJ+Nt+6choJUaN3MjljzlTAhntpQzuLH1zHP18ymW9cPsULlQWffZXN/OHdg6wqKKWzu5eLpqRx++JclkxO9fgvz/7DGlNjI/jaZZP57HnjPTp1RHOHk1++uY+n3jtEcoyD714znevnZFpDwASEgA33R9cX8bPX97LhW0tPeqLNDE1tSyfPfHCYP75fTE1LJ1PGxPKlRblcf07msE9aVzR28NBb+3g+3zWs8a6LJnL74lxiIrw3YGtHaSPf/csOPj7SwMK8FH58/Swmpcd6bX/GjISADfeyhnbeLarhH+eN90JVBqCzu4e/flzO7zceZHd5EykxDlYsmMDnF0wgLS7ijLZ1/LDGzy2YwD1LJ5ESe2bbGaqeXuXZDw/zX6/tod3Zw50X5nHP0slEOQJrhJUJHgEb7mbkqCrvH6jliY0HWbunivCQEK6bk8Hti3OZfpq5fPoPa2zqcLL87AzuvWIq45N9822rurmT/1y9m1VbS8lKiuJHy2dyybQxPqnFmOGwcDcedbCmlT+8e5AX8ktod/ZwwcQUbl+cy9Kp6YT0G3bY0+u6Ufmv3txHWWMHF01J49vLpjIz4/TDGkfC+/tr+f7LOyiqamHZzLE88KkZZCRGjdj+G9q6KK5t40h9G9PGxls3kTljFu7GKxraunj2wyM89d4hKpo6yEuN4bbFudx4bibv76/lwdf2sK+yZVjDGr2tq7uX371zgP9eV0iICN+4bApfXJTjkZO6vb1KVXMnxbWtFNe1uR5r2zhc18ahmlaaOrqPWf+y6WO4+6I85tmQXjNIFu7Gq5w9vazeXs4TGw/ycUkjjrAQurp7PTascSQcqWvjB6/sZO2eKqaNjePfr581qJB19vRS1tDOodo2DrvD+1BtG4frWjlc10aH85MboIeGCFlJUWQnRzMhJZqclBiyk6PJSIzizV2V/PH9Q9S3OZk7IYm7LszjsuljjvkmZMzxLNzNiFBVthTXs7KglBkZ8R4f1uhtqsobuyr54Ss7KWvs4DPzxnPfVdOIDA/lsLvlfbiujUPuEC+ubaO0oZ2e3k/+30SGh5CdHE12cgw5Ka4Qz05xPc9IjDrln0dbVzcv5Jfwu3cOUFLfzsS0GO66cCLLz8kgIsxO+poTWbgbcwZaO7t5ZG0hv994kBARunp6j3k/PjKMnNSYoy3wCSkxTEh2PabHRQy7td3d08vqHRX89u/72VnWRHpcBF9anMst87OJj7SLsMwnLNyNGYK9Fc08++FhUmIcZLu7USakRJMYPTJTM6gqG4tq+O3fD7CxqIbYiDBWzM/mtkW5jE2wSdGMhbsxfm9HaSO/3XCAv20rIzREuH5OJnddlDci00OY0cvC3ZgAcaSujcffOcD/5R+hw9lrI2yCnIW7MQGmrrWLp947ZCNsgpyFuzEB6vgRNnlpMdx1YR7Xn5NpI2yCgIW7MQFuoBE2ty3KZcUCz42w6elVGtq6qGvtora1i3r3Y12/n6YOJ+MSopiUHnv0Z1x8pH2b8BILd2OChKryblEtv92wn3cKTz3CpsPZc0wwnxjandS3Oqlt7aSutYuGdicni4i4yDBSYhzERIRR2tBOQ5vz6HvRjlDy0mKYlOYK+4nuxwkpMT6/85e/s3A3JggdP8LmvJxkWju7jwZ4a1fPgJ8LEddN20/8iSAlxkFSjIMU97KUGAeJ0Y5jQlpVqW3toqiqhaKqFvZXux+rWihr7Di6XmiIMCEl+mjYT0qLZWJ6LBPTYoiz8fyDYuFuTBDrG2Gz9UgDidGuQE6KdpASe2KAp8Q4iI8M91o3SmtnNweqWymqbnYHfitF1S0cqmmlu9+VvmPjI92t/BjXo7uLJy02YtRPZTGSLNyNMaOas6eX4tq2Y1r5RdWux/7fMOIjw5iYHktmYpT720MEybHHfpNIdn+bCIYbow823L13GxxjjDmF8NCQoydgr5z5yXJVpaKp44Qunh2ljdS2dtF83MyafUQgMSrcHfgRrm8m/X4J9F+eEuv6JhPI/f8W7saYUUVEGJcQxbiEKJZMTjvh/a7uXurbjj0hXNfS+clz92NRdQt1h7qob+s65Unh/t1TSdEOEqLCiY8KJz4yjLjIT57H91se4wgb9aOBPBbuIrIMeBgIBR5X1Z96atvGGNPHERbCmPhIxsQPbq6dnl6lsd1JXWsntS1dx/wSqOs3Sqikvp0dpU00dThpO8mJ5z4hgjv4w4iPDCcu0vXoCv9Plh/zi6Fvvahw4iK8/8vBI+EuIqHAo8DlQAmwWUReUdVdnti+McYMVWiIHG2dT0of3GecPb00d3TT3OGkqb2bpg4nTe1O92P/191HlxfXttHU4aS5o5uWzoG7jvr88LqZ3HpBzvAP7hQ81XI/HyhS1QMAIvIcsBywcDfG+J3w0JCjvxCGorunl5bO7pP+YjhvBOYF8lS4ZwJH+r0uAeYfv5KI3AncCZCdne2hXRtjzOgSFhpCYrRjxKaKHoinThUP1Hl0wikMVX1MVeep6ry0tBNPlBhjjPEMT4V7CTC+3+ssoMxD2zbGGHOGPHIRk4iEAfuAS4FSYDNwi6ruPMVnqoHiIe4yFagZ4mdHGzuW0SdQjgPsWEar4RzLBFU9bdeHR/rcVbVbRO4BXsc1FPKJUwW7+zND7pcRkfzBXKHlD+xYRp9AOQ6wYxmtRuJYPDbOXVVXA6s9tT1jjDFDF7jX3hpjTBDz13B/zNcFeJAdy+gTKMcBdiyjldePxWezQhpjjPEef225myAmIm+LSL2IRPi6FmNGKwt341dEJAdYgusiuetGcL82g6rxK34X7iKyTET2ikiRiNzn63qGSkTGi8h6EdktIjtF5Gu+rmk4RCRURLaKyKte3tUXgE3Ak8Ct/fYfJSK/EJFiEWkUkY0iEuV+b7GIvCciDSJyRES+6F7+tojc0W8bXxSR90XkRRHZIyIqIj8XkUKg0L3Ow+5tNInIFhFZctyfwb+KyH4RaXa/P15EHhWRX/Q/CBH5q4h83Xt/TCAi33D/29ohIs+KyOCmURwFROQJEakSkR39liWLyJsiUuh+TPJljYNxkuP4mfvf1zYReUlEEr2yc1X1mx9cY+j3A3mAA/gYmOHruoZ4LOOAc93P43BdBOaXx+I+hm8CfwZe9fJ+ioCvAHMBJzDGvfxR4G1c8xyFAhcAEUA20AzcDIQDKcAc92feBu7ot+0vApV9y3B9O1gPJANR7mWfc28jDLgXqAAi3e99C9gOTMU1JcfZ7nXPx3XFdoh7vVSgra92L/05ZQIH+9X9PPBFX/87OYP6LwTOBXb0W/ZfwH3u5/cBD/q6ziEexxVAmPv5g946Dn9ruR+dfVJVu4C+2Sf9jqqWq2qB+3kzsBvXf0i/IyJZwDXA417ez2JgAvC8qm7B9Yv+FhEJAb4EfE1VS1W1R1XfU9VOYAXwlqo+q6pOVa1V1Y9OsotIIAH4fb9lP1bVOlVtB1DVp93b6FbVX+D6BTLVve4dwPdUda+6fOxe90OgEdcV3ACfBd5W1UpP/dmcRBgQ5e5SisaPpgRR1Q1A3XGLlwNPuZ8/BVw/okUNwUDHoapvqGrfnMCbcE3X4nH+Fu4DzT7pl4HYn7sf+RzgA99WMmQPAd8Ger28n1uBN1S177LtP7uXpeIK5v0DfGb8SZYPJB3Xt4E/iMhW97JjLhEXkXvdXWmNItKA65dB6iD29RSuVj/uxz8NsqYhUdVS4OfAYaAcaFTVN7y5zxEwRlXLwdU4wvX35e++BKzxxob9LdwHNfukPxGRWGAl8HVVbfJ1PWdKRK4FqtwtaW/uJwr4NHCRiFSISAXwDVxdH+OADmDiAB89cpLlAK24WrR90oBY4Deqeo572Z39algCfMddR5KqJuJqkff9uzzVvp4GlovI2cB04C8nP9rhc/dHLwdygQwgRkQ+d+pPmZEkIt8FuoFnvLF9fwv3gJp9UkTCcQX7M6q6ytf1DNEi4DoROYSrm+wSEXnaC/u5HugBZgBz3D/TgXdwnWR9AviliGS4T2wudA+VfAa4TEQ+LSJhIpIiInPc2/wIuEFEokVkEq6upU5V7f8Nqt+tm4nD9Z+xGggTkQeA+H7vPw78WEQmi8tZIpICoKoluCbU+xOwsq+bx4suAw6qarWqOoFVuM5D+LNKERkH4H6s8nE9QyYitwLXAivU3fnuaf4W7puBySKSKyIOXH2Xr/i4piEREcHVt7tbVX/p63qGSlXvV9UsVc3B9fexTlW90UK8FfiDqh5W1Yq+H+DXuPrV78N1MnMzrj7OB3GdwDwMXI3r5GcdrkA/273NXwFduE6iPgX8EegUkamf7Jaifs9fx/UVeh+uGU07OLab8Je4Tly+ATTh+vuN6vf+U8BsvNwl43YYWOD+xSW4+vt3j8B+vekVPhkhdSvwsg9rGTJx3W/6O8B1qtrmtf146ZeG14jI1bj6ePtmn/yJj0saEvfJwXdwBVJfX/W/qmsCNr8kIhcD/6Kq1/q6lqFyt+ofxzUa6wBwm6rWe2jbF+LqnslRVW+fn0BEfgh8Bte3ja24RgF1enu/niAizwIX4zqfUQn8G66urOdxjYA6DPyjqh5/0nVUOclx3I/rRHyte7VNqnq3x/ftb+FujD9yd8E9B3ysqj/ydT0m8Plbt4wxfkdEpgMNuE78PuTjckyQOG24D3SF1XHvi4g8Iq4rRreJyLmeL9MY/6Wqu1U1RlUv8McRUcY/Dabl/iSw7BTvXwVMdv/cCfxm+GUZY4wZjtNOhqSqG9wX2ZzMcuCP7uE8m0QkUUTG9V1scDKpqamak3OqzRpjjDneli1banSE7qF6sqtGTwh3EbkT90Uh2dnZ5Ofne2D3xhgTPESkeDDreeKE6qCvGlXVx1R1nqrOS0sb8v2xjTHGnIYnWu4BddWoMcZ4Wm+vUt/WRWVTJ5XNHUxKi2V8cvTpPzgMngj3V4B7ROQ5YD6uCYpO2d9ujDGBQFVpbHe6Qrupg8qmDqqaj31e1dRJVXMHzp5POjR+vHwmn1+Y49XaThvu/a+wEpESXFdYhQOo6v8Cq3Fd3l2Ea47q27xVrDHGjARVpbmzm6qmDird4dwX4FV9Qe5e1tV94sXGCVHhjImPID0ukvl5MYyJj2RMXARj4iNJj48kLzXG68cwmNEyN5/mfQW+6rGKjDFmBHX39FJwuIH1e6soKK53t7o7aXf2nLBubEQY6fERjImLZG520tGwHhMf4Q7wSNLjI4gMD/XBkRzL7gtpjAk6tS2d/H1fNev2VLFhXzVNHd2EhQhnZSUwOyuRy462st2hHR9JelwEMRH+E5n+U6kxxgyRqrKzrIl1e6pYv7eKj440oAqpsRFcOXMsl0xLZ9HkVOIjw31dqsdYuBtjAlJLZzcbC12t87f3VlPV3IkInJWVyNcvncLSaWnMykggJGSg0dz+z8LdGBMQVJUDNa2s31PFuj1VbD5Uh7NHiYsM48IpaVwyNZ2LpqaRGhvh61JHhIW7McZvdTh7+OBgHevd3S3Fta57X0wZE8uXFudyydR0zp2QRHho8E2Aa+FujPErZQ3trN9bxfo9VbxbVEu7s4fI8BAumJjKHUvyWDo1jawk714g5A8s3I0xo1pju5MdpY1sLKph/Z4q9lQ0A5CVFMU/zsti6bR0FualjIrhh6OJhbsxZtRo6exmR2kj20sa2V7q+jlY0wpAWIgwLyeJf716GpdMS2diWiyu28OagVi4G2N8oq2rm51lTUeDfFtJAwdqWum782dGQiSzsxK4aW4WszMTmJOdGFBDFb3Nwt0Y43XtXT3sKm9iR2kj20oa2V7aQFFVC73uIB8TH8HszESWz8lkdlYCszMTgmZUi7dYuBtjPKrD2cOeima2lzS4W+SNFFa10ONO8tTYCM7KSuCqWeNcV4RmJpAeH+njqgOPhbsxZkjau3qoa+uiurmTXWVNbC9tYFtJI3srmul2B3lyjIPZmQlcPmMMszMTmJ2VwNj4SOsrHwEW7sYYOpw91Ld1UdfaRX2rk7q2LhqOvu6irs1JfWsX9W19r7vocB47G2JCVDhnZSVw54V5R4M8MzHKgtxHLNyNCUDdPb2UNrQ0wXraAAARsElEQVRTWt9OXV8gtzpd4dwX2m2uIK9v66Kt68QZEPvER4aRHOMgKcbB2PhIpo+LJznGQWJ0OMnRDpJjHEwbG8/4ZAvy0cTC3Rg/papUt3RysLqVgzWtHKhp5UB1KwdrWjhc13bMzSH6xPUFdbSDtNgIpoyJIznaFdyu5eEkuQM7KcZBYlQ4YUF4dWcgsHA3ZpRr7nByqKaNAzUt7vD+5Kels/voeo6wEHJSopmUHsvlM8aSlxZDVlIUKTERJMWEkxjlwBFmQR0sLNyNGQW6uns5XNd6THgfcD9WN3ceXU/EdWVmbmoscyckkZsac/QnIzGK0ACd4dCcOQt3Y0ZYXWsXa3dXsqu86WiQH6lrOzrmGyA11kFuagxLp6aRmxpLbmoMeWkxZCdH22X2ZlAs3AOcs6eX8oYOO9nlY9XNnbyxq4I12yt4/0AtPb1KtCOU3NQYZmcmsPzsDPLSXCGekxpDQpRdiWmGx8I9gG06UMsDL+9gX2ULiyal8N2rZzAjI97XZQWNyqYOXttRwert5Ww+VEevQm5qDHdflMdVs8YxMyPefuEar7FwD0BVTR38ZPVuXv6ojMzEKL5y8UT+/OFhrvnvd7jp3Cz+5cqpjLErAr2itKGd13ZUsGZ7OVsO16Pqmlv8nksmc/XssUwdE2eBbkbEoMJdRJYBDwOhwOOq+tPj3s8GngIS3evcp6qrPVyrOY3unl6efO8QD71VSFd3L/90ySS+cvEkohyh3HXhRH69vpAn3zvEq9vKueuiPO68MI9oh/1+H67DtW2s2VHOmh0VfHSkAYDp4+L55mVTuGr2WCalx/m4QhOMRPXEsbDHrCASCuwDLgdKgM3Azaq6q986jwFbVfU3IjIDWK2qOafa7rx58zQ/P3+Y5Zs+Hxyo5YGXd7K3spmLpqTxg+tmkpsac8J6xbWtPPjaHlZvr2BMfAT3XjGVG8/NslEWZ+hAdQtrdlSwZkc5O0qbAI7Ol3LVrLHkDPBnb4wniMgWVZ13uvUG02w7HyhS1QPuDT8HLAd29VtHgb7O3ASg7MzKNUNV1dzBf67ew0tbS8lMjOK3n5/LFTPGnPSr/4SUGP5nxVzyD9Xx73/bzbdf3MaT7x7ie9dM54JJqSNcvX8prGxmjbsPve+GEedkJ/Ldq6ezbNZYxifb3X/M6DGYcM8EjvR7XQLMP26dHwBviMg/ATHAZR6pzpxUd08vf3y/mF+9uY/O7l7uWTqJry51dcEMxrycZF76ygX8dVs5D67Zwy2Pf8Bl09O576rpTEqP9XL1/kFV2VPRzJrt5azeUUFRVQsicN6EZP7tUzNYNmss4xKifF2mMQMaTLgP1AQ8vi/nZuBJVf2FiCwE/iQis1T1mJmFRORO4E6A7OzsodRrgM2H6vj+X3awp6KZJZNT+eF1M8lLO/NAFhGuOzuDK2aM4Q/vHuJ/1hdx5UMbWDE/m69dOpmUUTSftqqyv7qVQzWtiECICLgfQwQE12PfMgFCQlyPInL0M33rinB0mfT7jIjQ1OHkrV2VrNlRwcGaVkIE5uemcOvCCVw5c6xNT2v8wmD63BcCP1DVK92v7wdQ1f/st85OYJmqHnG/PgAsUNWqk23X+tzPXHVzJz9ds4eVBSVkJETywKdmcOXMsR4bfVHb0slDbxXy5w8PEx0eyj2XTOLWC3J8dtFMW1c37xXV8va+Kt7eW01JffuI7Ts0RLhgYgpXzx7H5TPG2I0jzKgx2D73wYR7GK4TqpcCpbhOqN6iqjv7rbMG+D9VfVJEpgNrgUw9xcYt3Aevu6eXpzcV84s399Hh7OHLS/K455JJXhvpUlTVzH+s3sO6PVVkJUXxnWXTuPascV4fwtfXOn97ryvMPzxYR1dPL9GOUBZNSuXiqWnMzEhAcH117FVF1fU5BXp7lV4FpW+5ex33uugnn+lVdV8R6v5Mv3UdocL83BSSYhxePV5jhsJj4e7e2NXAQ7iGOT6hqj8RkR8B+ar6inuEzO+AWFz/776tqm+capsW7oOzpbiO7/1lJ7vLm1gyOZUfXDeTiUPoghmKd4tq+Pe/7WZ3eRPnZCfyvWtmMHdCkkf3cbLW+eT0WC6emsbFU9OZl5NERJhdcm8MeDjcvcHC/dRqWlxdMC9uKWFcQiTfv3YGV83yXBfMYPX0Kiu3lPDzN/ZS1dzJNWeN475l04Y8MsTVOm/h7b3VJ22dXzQljawkG3lizEAs3P1UT6/yzAfF/Pz1vbQ7e7hjSR73LJ1ETIRvLzZq7ezmsQ0HeGzDAXp6ldsW5fCVpZMGNQeKtc6N8RwLdz+0pbieB17ewc6yJhZPcnXBjLZhiRWNHfz8jb2sLCghMSqcr182hVvmZxPe74YOp2qdXzAxlaXTrHVuzFBZuPuR2pZOHnxtD8/nlzA23tUFc/Xske+CORM7yxr5yd92897+WvLSYvjOsmmEiljr3Bgvs3D3Az29yp8/PMzPXttDW1cPty/J5Z8vmezzLpjBUlXW7q7iP9bs5kB1K4C1zo3xMk9OP2BOorunl+aObpo6nMc8Nnd009Te99z92Omkqf2T103u9bu6e7lgYgo/Wj7T7yaYEhEumzGGi6am8frOCpKiHdY6N2aUsHAfgKqyqqCUfZXNNHU4aXIH9tFgdgd3u/Pkd4zvE+0IJS4yjLjIcOIjw0iMdjA+Ofro63OyEz16IZIvhIeGcO1ZGb4uwxjTj4X7ANburuLeFz7GERZCvDuE4yLDiI8KZ1xCJHER4Udf9wV3XGQY8cc9xkaGHXOi0RhjRoqF+3FUlYfXFjI+OYp1915s4WyM8UuWXMdZv7eK7aWN3LN0kgW7McZvWXr1o6o89FYhWUlR3HBulq/LMcaYIbNw7+ftvdVsK7FWuzHG/1mCuakqD60tJDPRWu3GGP9n4e72933VfHykga8unYQjzP5YjDH+zVKMT0bIZCZGcdNca7UbY/yfhTuwobCGrYcb+MrSidZqN8YEhKBPMlXl4bf2kZEQyT/OHe/rcowxxiOCPtw3FtVQcLiB/2d97caYABLUaeZqtRcyLiGST8+zvnZjTOAI6nB/b38t+cX1fOXiiTaToTEmoARtuLuuRt3H2PhIPn2e9bUbYwJL0Ib7+/tr2Xyonv9nrXZjTAAK2nB/aG0hY+Ij+Iy12o0xAWhQ4S4iy0Rkr4gUich9J1nn0yKyS0R2isifPVumZ72/v5YPD9Zx90UTiQy3VrsxJvCcdj53EQkFHgUuB0qAzSLyiqru6rfOZOB+YJGq1otIurcK9oSH1+4jPS6Cm8/P9nUpxhjjFYNpuZ8PFKnqAVXtAp4Dlh+3zpeBR1W1HkBVqzxbpudsOlDLpgPWajfGBLbBhHsmcKTf6xL3sv6mAFNE5F0R2SQiywbakIjcKSL5IpJfXV09tIqH6eG3CkmLi+CW+dZqN8YErsGE+0B3btbjXocBk4GLgZuBx0Uk8YQPqT6mqvNUdV5aWtqZ1jpsHx6s4/0Dtdx1YZ612o0xAW0w4V4C9B9SkgWUDbDOy6rqVNWDwF5cYT+qPLx2H6mxEayYP8HXpRhjjFcNJtw3A5NFJFdEHMBngVeOW+cvwFIAEUnF1U1zwJOFDtfmQ3W8W1TL3RflEeWwVrsxJrCdNtxVtRu4B3gd2A08r6o7ReRHInKde7XXgVoR2QWsB76lqrXeKnooHn6rkNRYh7XajTFB4bRDIQFUdTWw+rhlD/R7rsA33T+jzpbiOjYW1fCvV0+zVrsxJigExRWqD71VSEqMg88tsFa7MSY4BHy4bymu553CGr58YR7RjkF9UTHGGL8X8OH+8NpCkmMcfN5a7caYIBLQ4b71cD0b9lXz5SV5xERYq90YEzwCOtwfXltIUnQ4X1horXZjTHAJ2HD/6EgDb++t5g5rtRtjglDAhvsjawtJjA7n1gtyfF2KMcaMuIAM94+PNLBuTxVfXpJHrLXajTFBKCDD/ZG1hSREWV+7MSZ4BVy4by9pZO2eKu5YnEtcZLivyzHGGJ8IuHB/eG0h8ZFh3Loox9elGGOMzwRUuO8obeSt3ZXcsSSPeGu1G2OCWECFe1+r/YvWajfGBLmACfedZY28uauSLy3OtVa7MSboBUy4P7K2kLjIMG5blOvrUowxxucCItx3lzfx+s5KbluUS0KUtdqNMSYgwv2RtYXERYRxu7XajTEGCIBw313exJodFdy2KIeEaGu1G2MMBEC4//e6QmIjwvjSYmu1G2NMH78O970VzazeXsEXL8ghMdrh63KMMWbU8Otwf2RdITGOUG63VrsxxhxjUOEuIstEZK+IFInIfadY7yYRURGZ57kSB7avspnV28v54qIckmKs1W6MMf2dNtxFJBR4FLgKmAHcLCIzBlgvDvhn4ANPFzmQR9YWEh0eyh2L80Zid8YY41cG03I/HyhS1QOq2gU8BywfYL0fA/8FdHiwvgEVVjbzt+3lfOECa7UbY8xABhPumcCRfq9L3MuOEpFzgPGq+uqpNiQid4pIvojkV1dXn3Gxff57XRFR4aF8eYm12o0xZiCDCXcZYJkefVMkBPgVcO/pNqSqj6nqPFWdl5aWNvgq+ymqauGv28r4wsIckq3VbowxAxpMuJcA4/u9zgLK+r2OA2YBb4vIIWAB8Iq3Tqqu2V5OZFgoX15iI2SMMeZkBnOD0c3AZBHJBUqBzwK39L2pqo1Aat9rEXkb+BdVzfdsqS7/dOlkrj8nk5TYCG9s3hhjAsJpW+6q2g3cA7wO7AaeV9WdIvIjEbnO2wUOZHxytC92a4wxfmMwLXdUdTWw+rhlD5xk3YuHX5Yxxpjh8OsrVI0xxgxMVPX0a3ljxyLVQPEQP54K1HiwHF+yYxl9AuU4wI5ltBrOsUxQ1dMON/RZuA+HiOSrqtenOBgJdiyjT6AcB9ixjFYjcSzWLWOMMQHIwt0YYwKQv4b7Y74uwIPsWEafQDkOsGMZrbx+LH7Z526MMebU/LXlbowx5hQs3I0xJgD5XbgP9q5Qo52IjBeR9SKyW0R2isjXfF3TcIhIqIhsFZFTTvs82olIooi8KCJ73H83C31d01CJyDfc/7Z2iMizIhLp65oGS0SeEJEqEdnRb1myiLwpIoXuxyRf1jgYJzmOn7n/fW0TkZdEJNEb+/arcB/sXaH8RDdwr6pOxzWT5lf9+FgAvoZr7iF/9zDwmqpOA87GT49JRDJx3RltnqrOAkJxTfrnL54Elh237D5grapOBta6X492T3LicbwJzFLVs4B9wP3e2LFfhTuDvyvUqKeq5apa4H7ejCtEMk/9qdFJRLKAa4DHfV3LcIhIPHAh8HsAVe1S1QbfVjUsYUCUiIQB0Rw7VfeopqobgLrjFi8HnnI/fwq4fkSLGoKBjkNV33BPyAiwCdc06h7nb+F+2rtC+SMRyQHOYYTuP+sFDwHfBnp9Xcgw5QHVwB/cXUyPi0iMr4saClUtBX4OHAbKgUZVfcO3VQ3bGFUtB1fjCEj3cT2e8CVgjTc27G/hfsq7QvkjEYkFVgJfV9UmX9dzpkTkWqBKVbf4uhYPCAPOBX6jqucArfjHV/8TuPujlwO5QAYQIyKf821Vpj8R+S6u7tlnvLF9fwv3090Vyq+ISDiuYH9GVVf5up4hWgRc574L13PAJSLytG9LGrISoERV+75BvYgr7P3RZcBBVa1WVSewCrjAxzUNV6WIjANwP1b5uJ4hE5FbgWuBFeqli438LdyP3hVKRBy4ThC94uOahkREBFff7m5V/aWv6xkqVb1fVbNUNQfX38c6VfXLFqKqVgBHRGSqe9GlwC4fljQch4EFIhLt/rd2KX56crifV4Bb3c9vBV72YS1DJiLLgO8A16lqm7f241fhfrK7Qvm2qiFbBHweV0v3I/fP1b4uyvBPwDMisg2YA/yHj+sZEve3jxeBAmA7rv/rfnP5vog8C7wPTBWREhG5HfgpcLmIFAKXu1+Paic5jl/juvf0m+7/9//rlX3b9APGGBN4/KrlbowxZnAs3I0xJgBZuBtjTACycDfGmABk4W6MMQHIwt0YYwKQhbsxxgSg/w9y5SR89tFChwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time for GD is:\t2.937140941619873\n"
     ]
    }
   ],
   "source": [
    "# SGD 算法\n",
    "import time\n",
    "input_dim = 784\n",
    "num_classes = 2\n",
    "mini_batch = 8000\n",
    "model2 = LogisticRegression(input_dim, num_classes, lr=1000, batch=1000, weight_scale=4e-2, weight_decay=1.0, reg=0.0)\n",
    "\n",
    "tstart = time.time()\n",
    "model2.train(train_data[0:mini_batch], train_labels[0][0:mini_batch])\n",
    "tend = time.time()\n",
    "\n",
    "# 画图\n",
    "Loss, Acc = model2.history()\n",
    "plt.figure\n",
    "plt.subplot(2,1,1)\n",
    "plt.plot(Loss)\n",
    "plt.title('Loss')\n",
    "plt.subplot(2,1,2)\n",
    "plt.plot(Acc)\n",
    "plt.title('Accuracy')\n",
    "plt.show()\n",
    "print('Time for GD is:\\t%s' % (tend - tstart))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 计算参数\n",
    "分别在测试集上计算 confusion matrix, precision, recall, F1-score 等参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "# test 也需要归一化，参数就使用训练集中的参数而不是test自身的参数\n",
    "# 使用自身参数会导致已知了 test 部分信息\n",
    "test_data = (test_data - mean) / (std + 1e-7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy:\t0.9673469387755103\n",
      " precision:\t0.9654471544715447\n",
      " recall:\t0.9693877551020408\n",
      " F1-score:\t0.9674134419551933\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>[prediction]</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>[label]</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>950</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>34</td>\n",
       "      <td>946</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "['prediction']    0    1\n",
       "[label]                 \n",
       "0               950   30\n",
       "1                34  946"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 计算 GD 的这些参数\n",
    "import pandas as pd\n",
    "pred_y = model1.test(test_data)\n",
    "\n",
    "tp = 0\n",
    "tn = 0\n",
    "fp = 0\n",
    "fn = 0\n",
    "\n",
    "# 认为 0 是正例，1 是反例\n",
    "for i in range(len(pred_y)):\n",
    "    if pred_y[i] == 0 and pred_y[i] == test_labels[0][i]:\n",
    "        tp += 1\n",
    "    elif pred_y[i] == 1 and pred_y[i] == test_labels[0][i]:\n",
    "        tn += 1\n",
    "    elif pred_y[i] == 0 and pred_y[i] != test_labels[0][i]:\n",
    "        fp += 1\n",
    "    elif pred_y[i] == 1 and pred_y[i] != test_labels[0][i]:\n",
    "        fn += 1\n",
    "\n",
    "# 计算相应参数\n",
    "accuracy = (tp + tn) / (tp + tn + fp + fn)\n",
    "precision = tp / (tp + fp)\n",
    "recall = tp / (tp + fn)\n",
    "F1_score = 2 / (1 / precision + 1 / recall)\n",
    "\n",
    "confusion_matrix = pd.DataFrame(\n",
    "    {\n",
    "        '0': [tp, fp],\n",
    "        '1': [fn, tn]\n",
    "    },\n",
    "    index = ['0','1']\n",
    ")\n",
    "confusion_matrix.columns.name = ['prediction']\n",
    "confusion_matrix.index.name = ['label']\n",
    "print(\n",
    "    'accuracy:\\t%s\\n' % accuracy,\n",
    "    'precision:\\t%s\\n' % precision,\n",
    "    'recall:\\t%s\\n' % recall,\n",
    "    'F1-score:\\t%s\\n' % F1_score\n",
    ")\n",
    "confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy:\t0.9566326530612245\n",
      " precision:\t0.9765708200212992\n",
      " recall:\t0.9357142857142857\n",
      " F1-score:\t0.9557060969254819\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>[prediction]</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>[label]</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>917</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>22</td>\n",
       "      <td>958</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "['prediction']    0    1\n",
       "[label]                 \n",
       "0               917   63\n",
       "1                22  958"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 计算 SGD 的这些参数\n",
    "import pandas as pd\n",
    "pred_y = model2.test(test_data)\n",
    "\n",
    "tp = 0\n",
    "tn = 0\n",
    "fp = 0\n",
    "fn = 0\n",
    "\n",
    "# 认为 0 是正例，1 是反例\n",
    "for i in range(len(pred_y)):\n",
    "    if pred_y[i] == 0 and pred_y[i] == test_labels[0][i]:\n",
    "        tp += 1\n",
    "    elif pred_y[i] == 1 and pred_y[i] == test_labels[0][i]:\n",
    "        tn += 1\n",
    "    elif pred_y[i] == 0 and pred_y[i] != test_labels[0][i]:\n",
    "        fp += 1\n",
    "    elif pred_y[i] == 1 and pred_y[i] != test_labels[0][i]:\n",
    "        fn += 1\n",
    "\n",
    "# 计算相应参数\n",
    "accuracy = (tp + tn) / (tp + tn + fp + fn)\n",
    "precision = tp / (tp + fp)\n",
    "recall = tp / (tp + fn)\n",
    "F1_score = 2 / (1 / precision + 1 / recall)\n",
    "\n",
    "confusion_matrix = pd.DataFrame(\n",
    "    {\n",
    "        '0': [tp, fp],\n",
    "        '1': [fn, tn]\n",
    "    },\n",
    "    index = ['0','1']\n",
    ")\n",
    "confusion_matrix.columns.name = ['prediction']\n",
    "confusion_matrix.index.name = ['label']\n",
    "print(\n",
    "    'accuracy:\\t%s\\n' % accuracy,\n",
    "    'precision:\\t%s\\n' % precision,\n",
    "    'recall:\\t%s\\n' % recall,\n",
    "    'F1-score:\\t%s\\n' % F1_score\n",
    ")\n",
    "confusion_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SGD 和 GD 的比较\n",
    "从实验上看 GD 的用时比较长 iteration 也比较多，我认为用时长是可以理解的，因为每一个 iteration 的计算量都很大，但是 iteration 比 SGD 少可能是因为 SGD acc > 0.97 是在 batch 上而不是在 整个训练集上；如果要使得 SGD 也在训练集上 acc 上的话我们需要计算两次 logistc forward pass, 这显然对比较时间上是更不公平的，所以我选择了前者。另外一个可能的原因是训练集比较小，需要迭代的次数相对比较小。\n",
    "\n",
    "评估 GD 和 SGD 的几个参数差不多，两者性能相差不是很大，加上对时间因素的考虑，还是使用 SGD 较优。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 对数据集重采样消除不平衡因素\n",
    "从上面的 confusion_matrix 我们知道测试集上 0 和 非0 的样本数量是一致的，而在训练集上他们的分布是不平衡的，这种分布的不一致性会影响分类器的性能，所以下面我们在数据集上进行重采样，观察重采样训练集训练出来的分类器的表现。同样因为显存有限，我们采 4000 个 0 和 4000 个非零的样本作为新的数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4000\n",
      "4000\n"
     ]
    }
   ],
   "source": [
    "# 重采样\n",
    "import random\n",
    "index_0 = []\n",
    "index_n0 = []\n",
    "\n",
    "# 训练集随机化\n",
    "indexs = list(range(len(train_data)))\n",
    "random.shuffle(indexs)\n",
    "for i in indexs:\n",
    "    if len(index_0) == 4000 and len(index_n0) == 4000:\n",
    "        break\n",
    "    if train_labels[0][i] == 0 and len(index_0) != 4000:\n",
    "        index_0.append(i)\n",
    "    elif train_labels[0][i] == 1 and len(index_n0) != 4000:\n",
    "        index_n0.append(i)\n",
    "\n",
    "print(len(index_0))\n",
    "print(len(index_n0))\n",
    "\n",
    "# 构成数据集\n",
    "index = index_0 + index_n0\n",
    "random.shuffle(index)\n",
    "\n",
    "new_train_data = train_data[index]\n",
    "new_train_labels = train_labels[0][index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAACRCAYAAADTnUPWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAHt9JREFUeJztnXmQHdV1xr/z3rxZNIs0WtGGEJZACBMMKBgcx6bAUEBVCuyUU17i4DIOLu97GZzNFVc5TrliOxU7cUiZgCsux6SwAzbeiAzGBNugmFUIgViENjSMpNFomeUtN3/M89xzTk/fee/NW/r1nF/V1HS/2933vj7d9/X9+txzyDkHwzAMo/3JtLoBhmEYRn2wDt0wDCMlWIduGIaREqxDNwzDSAnWoRuGYaQE69ANwzBSQqo7dCJ6kYjeVOG2jog21FhPzfsa1WN2TSdm17mT6g49iRBRFxHdQkSjRPQyEX2i1W0y5g4R/QkRPUhEJ4novla3x6gP7Xa/drS6AfOQzwHYCGAdgFMA3EtETznnftLSVhlz5TCArwLYBODSFrfFqB+fQxvdr/PmCZ2ILiSiXxHRCBEdIKKvEVGn2uxqInqeiIaJ6EtElGH7v4eIdhDRESL6KRGtq7Epfwbg8865I865HQD+DcC7azzWvCcpdnXO/Y9z7nYA++fyfYwpkmJXtNn9Om86dABFAB8HsBTAxQAuA/ABtc2bAWwBcD6AawC8BwCI6FoAnwXwFgDLAPwSwHdmqoSI3kFEj8eUDQJYBeAx9vFjAM6u6RsZQALsajSEltu1Le9X51xq/wC8COBNMWUfA/B9tu4AXMnWPwBga3n5xwCuZ2UZACcBrGP7bqigPWvL23azzy4H8GKrz1U7/SXNrqr+9wK4r9XnqB3/kmbXdrxf580TOhGdQUQ/LL/YGAXwBUz9+nP2sOXdmPp1Bqb0s38sD/9GMKWXEoDVVTbjePn/APtsAMCxKo9jlEmIXY06kxC7tt39Om86dAD/AuBpABudcwOYGpKR2mYtWz4VXg/dA+B9zrlF7K/HOfdgNQ1wzh0BcADAuezjcwFsr+Y4hqDldjUaQsvt2o7363zq0PsBjAI4TkSbALx/hm0+TUSDRLQWwEcBfLf8+TcA3EREZwMAES0korfW2I5vAfjLcj2bAPw5gFtrPJaRELsSUZaIujHlOZYhom4iytVyLANAQuyKdrtfW635NPIPTJMD8AZM/eIfx9RLkr8F8ADb1gH4CIDnARwC8A8Asqz8XQCewNRFtgfALWrfDeXldwLYHmhTF4Bbysc5COATrT5P7faXULu+u7w9/7u11eeqnf4Sate2ul+p3GjDMAyjzZlPkothGEaqsQ7dMAwjJViHbhiGkRLm1KET0ZVEtJOIdhHRjfVqlNFazK7pxWybbmp+KUpEWQDPYGrm1F4ADwN4u3Puqbh9OrMLXE9uIK7YaBJj+VFMFk9qn14Atdm1o7vXdfUtbkhbjcqZOH4YhfETM9oVqN62uc5e19092JC2GtVx/Ni+Yefcstm2m0u0xQsB7HLOPQ8ARPSfmIqnEHvj9+QGcPG66+ZQpVEPfrX7tlBx1Xbt6luMs/7o43Vto1E9O37wldk2qcq23d2DuOCiD9W1jY2A5uCo52J//pLFL+65aXcl281FclkNOfV2L2aYWktENxDRNiLaNlkcm0N1RpOo2q6F8RNNa5wxJ2a1LbdrPm92bTfm0qHP9NsW+a10zt3snNvinNvSme2ZQ3VGk6jarh3dvU1ollEHZrUtt2suZ3ZtN+YiueyFjKWwBu0UC7rVE6oosWO9RNi15mF0q+fJ1WjWJg39m2LbuUggFVOnOoJtbYBNGm3nuTyhPwxgIxGtLweefxuAu+rTLKOFmF3Ti9k25dT8hO6cKxDRhwD8FEAWU7ESEhuFzKgMs2t6MdumnznlFHXO/QjAj+rUlvpQqxtmqcL96iXVBCQXl6liXNYA6aZZdq1qaM62JW0DURbar4qyaprGTaDM4TIx2+lttR0DZp3LsL1etq2brBI4TsTOMfsFtytV36RpAvqF4/bSNg+Uhc5bPeQYmylqGIaREqxDNwzDSAnWoRuGYaSEOWnoiSCgnwV18YgOW8W2nFAdIS08opn6dQrofhF9nbctua6Q0wS115Ceys6zPgY/X/rcif1CZcVAuxROPwax017KUuy2LlCGjPxSDvE6bFujbcev31CZvicct2vonUrtYr/QwvV9xy/CbPz7j+D7sMAha9XT7QndMAwjJViHbhiGkRLaQ3KpRlYRQ7hAWTX7FYusqPIhHAVkFWTUbylf18M0VhaWYyINmL2RDaYaiYVcQB5hkkim6OLL8rIsU3CsrKTK/DoVVGP09REYRrucP/GlnDRCKee3LXaqMnb3ReUYx5ZlU/j10czgUrW6Kop7NCSraLtySaxQqrgMJWbXYuA+Vzh1T46t659ePnqazPc9yYLGbv/wP4uyM27z+azX/WRc1cFsF9Fc2GKN59qe0A3DMFKCdeiGYRgpwTp0wzCMlNAeGrpCanIBjaxYCpRJPzXHdDddJo6j9wtp6ln1e5nN+uVMVhQR37ZDmYV/X33MgLuj0F4ToKcDkC5lJa2Z+uWMMoHQwieVTs608eyk0skn/IEyYwVZNj7p657Ii7KRC1aI9YOvZed5ldRF33H2b6aXL+uXoVEu7vL1n/uND4uyZY/69pSkRItSB7FlWeZa5NHoKtR4g26EkfcfTO/W10Pen7uIhs7KkC+oskJsGdfXNcNXrBfro6f7Lzy5Xtp81fKR6eUvHX6VKNtwkc9FUfjhclkJvydlFyDfzZjbomEYxvzGOnTDMIyUkFzJJTgTLORiyIZUenjF3Q+1HFPwQzNXUMO0vB+O6/0yC/vFOnV3+5WA5OJy8tRz6YZGj8v9uAQTkniSIqsEEK6J6qtwmSWj3Ai5zKJlleyY3zE7rmSVExO+vpNy2Dx0qc++dmSzGu4rWSWb9XV2dso67n35jOnlvWMyqfL+QS/BXHHtQ6LssYfPm17WronE1qlEqoy5NDbR5hW7oGrXRCazkLp/uJRC+XhZhbSsMuHlMvBlAG5yki1LKa3wmg1ifWiLz6JW6EYsmaFOsb4v75Oi3164QJQdfXLJ9PL6ok7jl5lxEZjBjbEG7AndMAwjJViHbhiGkRKsQzcMw0gJydHQq4mKFnJNDLgfCm1c6eQuH6+hZ1ev9Ifvk0Jboa9LVt/FdHKlbzrhiqanejO3uOJiUdb76B6/ojVTPmU8cg5bo6lTQE+VLmxqP6a1Rqbwc9fEMbljdszrpFwz1zz9kVVyv5Un/UpJTcsvynNXennB9HLfTnVeD/ZNL+4qLJPt/iv/PfacWCTL2Hdy6n2LCAUQOIeNJlY3D4VtUNehjHgZcFvMS7sS079J6+TjzM5KJz/5Oq+T62vs8FnSP5S/u8ioV2fgr1Ei97LvOg/l+kTZot1sRb//E1E11bsRN/d3I/aEbhiGkRKsQzcMw0gJyZFcFBXPBtUwySXimsjXl0pZo7jch08rLJCnpW4JcTk6uB+LypdfKH9ne5b5oXrmiHJpTDjR4TcvQ3xZUUsuXI5RQ3PmquiycvrdzvctnF7uOkW6kJWYO2DxiByKr/uBrH/B/z7l95uQsg51epc2d+Y6UfaLF/wsws+c+zNR9ndXXTu9vObnVWTYSCL8dIWSUURmivIyZVfuqlhQ52eRv1+PXLBUFB1f4++fekWjJCXBZZgCVMjL+3XkLH8CBndKd0ftVltv7AndMAwjJViHbhiGkRKsQzcMw0gJidXQKyakry+S0/ILTCefHJCaKc+Co3U37kI0mybHdWHtohXaV7p9yQ1fudBPJ1+8XbpJ5g4cYQdJyNT/kNui2E7pqaGMRaHsNUxrfea90m2wb62PilcoSH0985S/PjbdcViWDR8R667Ln/dMJGEwy1ikypYs9Lp9f3ZMlPVuOOpX7pWub+IS0GZNgJmDyZcj92SgTLgZ63AcXjc/ce5qUfTKa1R4Sl5FdublmeBujdUkCg8ec7EX2E+eIu/X/t3xldQjjIM9oRuGYaSEWTt0IrqFiIaI6En22WIiuoeIni3/Hwwdw0geZtf0Yradv1QiudwK4GsAvsU+uxHAVufcF4noxvL6Z+rfvNmZ2CgTEZxcztyE1AgmdyI+KbBI3qqGaXxWp57B2DkiZ7AFk9cyJhfJoVixx5uipBIG5/tYoP1B6QaVe7nmYdqtSLBdNcLdTbm+FQf9LM6lZ78iy9gM0MxW2Yetvnv/9LKeqYkFPWKVz1rUUTQPX+oTIxx9i3QrvWTpvunlkWKvKOv6b++O6iLTZv1iJEn07NyKdrEtv0WU5DC6xcsshzarm5Kdk1JOXQ+dM28HRGWVTJ5vqrVWtpiVdZQ62H2eU/f5kL+3B3YpV9nO+Jnk9WDWS8U5dz+Aw+rjawDcVl6+DcC1MNoKs2t6MdvOX2rV0Fc45w4AQPn/8rgNiegGItpGRNsmi2NxmxnJoCa7FsZ1zGcjgVRkW27XfN7s2m40/KWoc+5m59wW59yWzmzP7DsYbQG3a0d37+w7GG0Bt2suZ3ZtN2p1WzxIRCudcweIaCWAoXo2qhr2v05GP1zwste2ukZ11uT44zihyUltq2e/H1l0DB0VZe7wiFzn05d1sml+zKVLxHr3gL95Rs+SWu/JFV53Gzpfumut38Hq1u501VOzXcU0/kpd7iIaYmXhHkhlohrZ4N0PieRTZW+nf8dxTCWX1lmjBMqFrrjSh4p47o+li+HA2Yeml0/tlfVPFH0d//qVa0TZwr2+bcUeqRFzt8WQG20V1PWe1W0QboxV2JXvd+hC6XI6sskvZ+SrKqFpF+XrKJQ6fVnfS/KZtftQIKSEfufFvsfhTSoaJtPNc10qm9IaL8wfWy9/FHv3sRCODXA/rfUJ/S4A15WXrwNwZ32aY7QYs2t6MdvOAypxW/wOgF8BOJOI9hLR9QC+COByInoWwOXldaONMLumF7Pt/GVWycU59/aYosvq3BZZr0j4IIdJL/zpWl+mRkk80XDHmNyve9gPdyYWqxmXJ/ywKffAc7HtKkWSUqskGnxdB/Nn7m5u9Jg8zrhv28LDUtY5+iofsW98eRXT2QJD83rblcsCXH6JDM1rzM4gZt2q62HwUS97vfRmKcGtWjQ6vTwxqM4Hl26U/LLvCpkMo3ipryNXlLYrMdfIXY+slWXf8zLC4oxMPK1llnrRknu2UvkgI58hh17npayRM6Vds2P8opKHEdebuiXW3+n1mc798l6Ckkh5cgyn5aBOL28ePX2TKHPMVbKrSybYeP3qF6aX79twvijr3Yd46iDB2ExRwzCMlGAdumEYRkqwDt0wDCMltEW0xeI3pVvQ+BNe+9RuSVxP69t+UJSVhv3kuV6Vscgd81O2tZZWceJpAI5P/Y+4EVb2+zm5Ueq3XHbO5LWYWGHUu4REYgy548kE0vER+7TbIp302nTn1jWirPQWr3evvvwlUfZC96nTyxOnySxEHd1yCr8b8/PJczsWiLKBR/w1sHKn9AYsLfTbFnviIwRGky7Hb5oIArYLMXKmdOM78gZvOxqS77VE5FKlkw8wUy55ZFSUZfYyG+j7c0xObuT3L3Wo7pCtR67VDhZGRBXlS/7diH7H12jsCd0wDCMlWIduGIaREpIjuWhJgMkFP9l0tyha//QN08vLHpUuQ12P+7GYlkM4paHhipvGj+Py6pilQMB6SLc08Q2VHJPf5N3djpwpXe9k8uR46SSSUGPuM0fnTmCmaDBJtC7TyQ9iWHX3HrG+c72XYE45R0pwmXO8SxuNySiWhUMyTMWpP/T199z/mGxbD7PX4EJRFpTEQnC5IZAvoh3gs0EP/qG6Xwr+mTKrvye7VjLyNsfyHz3vt1PXhmMuwG5STjF1KsG36HdUgvGoZMrgCcZL8rl4rOiltUiiFu7SG3/0mrEndMMwjJRgHbphGEZKsA7dMAwjJSRHQw+wfVK6GmVP+t+h7qekK1pIlyKe7FlNQQ65Jop1N4uWy+qgrM6ywqb+n7pSFJ1Y43XYSPIaXmUbCKhxYQCmCkM7Vridft/CzmtxsYyEWFzo33lMFOTlns+z7DHq3cTp/yVF2+x9v/UrC6TbIgn3toDuGnGH5YnB9RdOwPuPesG+59LVcir+8JBP3B75yuwWjUSc5OdLv8di92vknZeC36MRt8UlPuppSb5iERp6oSD7kge3nTm9vOF+FVOe6/Khe6NG89sTumEYRkqwDt0wDCMlWIduGIaREtpCQ//12Hqxzqe/l5YtEmV04BDbMD7mpvYPFZq61jO5zqba5pTvd6bX66uZxbJtJ84+xbe7Mz7DuCY77gvX33kyfsOEQJVq4TpcaUhTZrZ0WXnuCiu87/e+T0nN9MzB/dPLz+xdIes75IXR7AoZ2nb496ROvvL/WFakwBTxiL4f8K3nhLX3+KJEomz30lX+3PWTPM/8pJTUaSW27nQZfx+lQzQze2Q6lY+6OpDYdpnMIrb3Kn+9FHrjJwMU8vKYyx/iob9V/ZnGhEz+HfaEbhiGkRKsQzcMw0gJyZFcAsPvHw+/WpRlJ/yQZvgCmVC574Af3vVse16UgdjvV0ZF7GOrTk8B5s189YbY/QCg2O1P6fhC5etUqSuS2m75r300ueywysCSba/fZBFBT3uAVihJFBbKafmlv/Ey20V9h0XZI0Orp5fP+CfpipgZ8dlrdv61nLKff6M8z8f3bp5e7v+lvq4qM2wkYh+XBEPJtFtIrAtqwOVubIWMmthzgbdPRulOxOb788TLgMwEteoBNWWfSSWhwI96+n7hnNPF+sHf99dSSQXDzPdVpnUt/4H8vgM7fYRP1xWf/LsRtFdvYBiGYcRiHbphGEZKsA7dMAwjJSRHQw/wwojMLkTMM01PyT25zH+liTdtFGWZAnOL02Etufuh9pjLV7gfatfIeoa9vrvoSZkth0ZYlvkO5faUkExEFRMIC1upu+Pwp2QoiCuW+PAPwxNy6n/vv3vX0cwRmU2Ixn1oVVeS5zGblYYeOd2f9/5fxrctAn9kUrYS2ZsS+mhVS+ak/ZfI9fet92ETbn/xvPgd1TlwTF/PTKobj2njJ85bK4oKC/yB+L0LAIc2a/9HXp8qCn33jC/MTugOI7Bfg0noZWQYhmFUi3XohmEYKaEtJJfCz5eKdWIJYkISSLRs5u0AGeEwUhbYD3pbPqNR6Qa5UV9JbuiYKMNhlj1Hyyo6amMboWd8Viqr6P3GTvFGv3jVTlE2NOFdVXf9/WZR1v/kK/6YBRmVb2yTn7nb0SVdGgsFec5zfFcdcZO7xik3UsfWXYeSXAJui6GyJMLtlZmQDT5W9LYrFJUbH4tUSAU145Ot771UztzN5Fnybek1qJKqh9strsFAIFUXSKcUmdXMo7pqSbTBEqk9oRuGYaSEWTt0IlpLRPcS0Q4i2k5EHy1/vpiI7iGiZ8v/B2c7lpEczK7pxOw6v6nkCb0A4JPOubMAXATgg0S0GcCNALY65zYC2FpeN9oHs2s6MbvOY2bV0J1zBwAcKC8fI6IdAFYDuAbAJeXNbgNwH4DPNKKRa76/V37AtNBjF6wWRaWOeI2KikznK2gtnG2nJbGCL8xOSKGtY1RGj3NPPef3y8VHdkMuELFPZ1PiultAg3OhLOV62wTYtRqOrvfaa4dK6fT0EX8NLH7kgNyRZazRGd9ffu0avqEoK+Sl1rtkN6tTvdNwzJauU9q11Om3LakokTxqZKSMZl7WzJDJp652DWafikO1qT87PvN2AMAyRVFe7pid9OvFTuUezK71SIYvtp5RZRruLqpdR7uO+Dp696myEb/xgt3yfZjL8RAjOrsWb6hqTB3k9ao0dCI6DcB5AH4DYEX54vndRbQ8Zp8biGgbEW2bLI7NtInRYuZq18L4iZk2MVrMXO2az5td242KO3Qi6gNwB4CPOedGZ9v+dzjnbnbObXHObenM9sy+g9FU6mHXju7exjXQqIl62DWXM7u2GxW5LRJRDlMXx7edc98rf3yQiFY65w4Q0UoAQ/FHqDNMduh/SCWJ7mcXoY5EyIbfOvA8l3EiCS540tlxOWwvHVPuh3w4rt0NucyikiTwgP0RySUTkFxCblCzuEi12q56iCuHv7LtPcPeJgU1pU+4wpFMcOFO+KdMt26VKBs/w0sBGeVOl3tGPnwMPOilNHRLP7kSWy/2yJB9QnLplF+4lONJy0WRXK9yaF5Pu9YyU3TFr+VOt5/hZ4f2d8v7ZzTL7teI1OmXM0qO4e6IEffkkMwSeIRdfb9KRj/qZxJnJqQkx6U1l1MSXAdPviHrCCYyqQOVeLkQgG8C2OGc+zIrugvAdeXl6wDcWf/mGY3C7JpOzK7zm0qe0P8AwLsAPEFEj5Y/+yyALwK4nYiuB/ASgLc2polGgzC7phOz6zymEi+XBxA/yLusvs0xmoXZNZ2YXec3bTH1P6IFc31V6c006iMVuoLUU8HWS3mltfJt9fR+PtWbVH06YXDOa6jUqVKg8CwrWifnenvE1Ylt2wbRFYVuqCMMsih1WjfmmnKpIAuL7FSu6JTv+NYOHJlefvqdMsLmmnv9/Jnn39wtyjo6472uljyphFimk5cG5DT0Yi/X0OX1UOz230MnBuevAnTi63aY7s/h2vCCA1In7/2Cfx+x6/1Sb16w0Nvg5Lh8Cesm2baBKfwRnT8QxbL/JSm497/o688dlFmqXLcP5ardUbluXuqQlYj3PwG7NkJPt6n/hmEYKcE6dMMwjJSQWMlFzATTUdAoXnIRm+ljBsrEcUqBsGta4tGukVyC0cmmQ7NBAzJSxbND20GO4a6JajjKk/QWS/IcDOzxY+4nRqX74WVLn55eXnC1HJsfuGRgenlZXkpgE3kmgd0tk6gMsCiNAFAc9BEdSwvkcQo9bPjdpaQi5qpYUncb//7VuHA2k7hZqpGP+Qd61itbPvW78p7Y+w4va/StlIldjmd8shKXkftl+K2kojT2DPkalzwlZ6l27j4k1sUs3x7ljspllg7tcspcE7V9uF0jSU0qi6JZa6Ice0I3DMNICdahG4ZhpATr0A3DMFJCcjR0rf867t6mMpkg4MZXiteiiWva2t2QT/fXbosc3ZbQVHw99V+UNT6iYiLQZmUflFQWGCcyvaj92Pna//UNouzZT49ML//HafeJsofYlO2fHTtHlN1x86XTy0sfl4Go8sv7xTrXTENT+PkyAJQCromhSH8hl7Za9dVGItqrHxNZWe6ofMdx+td4BFR1vxS8TSgv3UiJheMI3q/qPiv1SZdTro077X7YEa+T8+sxktw6E6+hNxp7QjcMw0gJ1qEbhmGkhORILho+VHGB4Pbaw1DMqlRDsdAMU1FBKHvxLEOoSt0IA1JNVbJKG7gqCsQsPn0O/Hkvaa2Gfc9MXtrnt58/f3r5jYXzRZlI4KvMupBNP5xc3CnKIu5mXDqJDL/jy4TkEpEi4uuTZUg+vL3KdsROvJY1hM319cC2pS7lAlysIQwkZpC9Am6EIZdgcZxqIio22Jb2hG4YhpESrEM3DMNICdahG4ZhpITkauicalwauaYe2C/yUxbSzatpW43bBnXzdtPJGSH9Nxqagb/jkPbg3qgU0bDZek7uR6UKz11EB9XlAZfKQHYhGXlSl4V02LiGJgPdPgrE1eB2JfUiQ9hO35PMdk7dn6Q8HCslZNeI+2HF0/TrEymzHja3J3TDMIyUYB26YRhGSmgPyUUTki74EC4ioyR4HNvGskqtRIbtvEy7l3FTBobbwaTGNapq0UrUYevgppZ0iWU2ePuDCSdC964+CRXKKlSrXKrrb4K7YaPtbE/ohmEYKcE6dMMwjJRgHbphGEZKIO0O1NDKiF4BsBvAUgDDTas4zHxsyzrn3LJ6HczsOitm1/oxX9tSkW2b2qFPV0q0zTm3pekVz4C1pX4kqf3WlvqRpPZbW8KY5GIYhpESrEM3DMNICa3q0G9uUb0zYW2pH0lqv7WlfiSp/daWAC3R0A3DMIz6Y5KLYRhGSrAO3TAMIyU0tUMnoiuJaCcR7SKiG5tZd7n+W4hoiIieZJ8tJqJ7iOjZ8v/BJrRjLRHdS0Q7iGg7EX20VW2pB2ZX0ZbU2NbsKtrSFnZtWodORFkAXwdwFYDNAN5ORJubVX+ZWwFcqT67EcBW59xGAFvL642mAOCTzrmzAFwE4IPlc9GKtswJs2uEVNjW7BqhPezqnGvKH4CLAfyUrd8E4KZm1c/qPQ3Ak2x9J4CV5eWVAHa2oE13Arg8CW0xu5ptza7ta9dmSi6rAexh63vLn7WaFc65AwBQ/r+8mZUT0WkAzgPwm1a3pUbMrjG0uW3NrjEk2a7N7NBnigQ8r30miagPwB0APuacG211e2rE7DoDKbCt2XUGkm7XZnboewGsZetrAOxvYv1xHCSilQBQ/j/UjEqJKIepC+PbzrnvtbItc8TsqkiJbc2uinawazM79IcBbCSi9UTUCeBtAO5qYv1x3AXguvLydZjSxhoKERGAbwLY4Zz7civbUgfMrowU2dbsymgbuzb5RcLVAJ4B8ByAv2jBi4zvADgAII+pJ5DrASzB1NvpZ8v/FzehHa/H1PD1cQCPlv+ubkVbzK5mW7NreuxqU/8NwzBSgs0UNQzDSAnWoRuGYaQE69ANwzBSgnXohmEYKcE6dMMwjJRgHbphGEZKsA7dMAwjJfw/IwjAzdOwun4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 观察新的数据\n",
    "plt.figure()\n",
    "plt.subplot(1,3,1)\n",
    "plt.imshow(new_train_data[26].reshape((28,28)))\n",
    "plt.title('label: %s' % new_train_labels[26])\n",
    "plt.subplot(1,3,2)\n",
    "plt.imshow(new_train_data[16].reshape((28,28)))\n",
    "plt.title('label: %s' % new_train_labels[16])\n",
    "plt.subplot(1,3,3)\n",
    "plt.imshow(new_train_data[5].reshape((28,28)))\n",
    "plt.title('label: %s' % new_train_labels[5])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 对新的数据集进行 GD "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iteration:\t0\t\t\tLoss:\t0.8603292346471882\n",
      "Accuracy:\t0.546625\n",
      "iteration:\t1\t\t\tLoss:\t0.6036549162660397\n",
      "Accuracy:\t0.721375\n",
      "iteration:\t2\t\t\tLoss:\t0.515378167988081\n",
      "Accuracy:\t0.784\n",
      "iteration:\t3\t\t\tLoss:\t0.46934100409027396\n",
      "Accuracy:\t0.811\n",
      "iteration:\t4\t\t\tLoss:\t0.4343711165356037\n",
      "Accuracy:\t0.842875\n",
      "iteration:\t5\t\t\tLoss:\t0.4057622372762437\n",
      "Accuracy:\t0.85975\n",
      "iteration:\t6\t\t\tLoss:\t0.381458331394844\n",
      "Accuracy:\t0.873625\n",
      "iteration:\t7\t\t\tLoss:\t0.36033748013746314\n",
      "Accuracy:\t0.88725\n",
      "iteration:\t8\t\t\tLoss:\t0.3416896091382821\n",
      "Accuracy:\t0.897875\n",
      "iteration:\t9\t\t\tLoss:\t0.3250286753606246\n",
      "Accuracy:\t0.907125\n",
      "iteration:\t10\t\t\tLoss:\t0.31000573827717437\n",
      "Accuracy:\t0.915125\n",
      "iteration:\t11\t\t\tLoss:\t0.29636246951182726\n",
      "Accuracy:\t0.921125\n",
      "iteration:\t12\t\t\tLoss:\t0.28390376297486886\n",
      "Accuracy:\t0.927375\n",
      "iteration:\t13\t\t\tLoss:\t0.27247846971471923\n",
      "Accuracy:\t0.93175\n",
      "iteration:\t14\t\t\tLoss:\t0.2619623736048855\n",
      "Accuracy:\t0.93675\n",
      "iteration:\t15\t\t\tLoss:\t0.25224750819478226\n",
      "Accuracy:\t0.941625\n",
      "iteration:\t16\t\t\tLoss:\t0.24324125219871723\n",
      "Accuracy:\t0.94525\n",
      "iteration:\t17\t\t\tLoss:\t0.2348666904513979\n",
      "Accuracy:\t0.949\n",
      "iteration:\t18\t\t\tLoss:\t0.22706028417752575\n",
      "Accuracy:\t0.952125\n",
      "iteration:\t19\t\t\tLoss:\t0.2197691642595772\n",
      "Accuracy:\t0.95475\n",
      "iteration:\t20\t\t\tLoss:\t0.21294892726398545\n",
      "Accuracy:\t0.956625\n",
      "iteration:\t21\t\t\tLoss:\t0.2065612950918989\n",
      "Accuracy:\t0.958375\n",
      "iteration:\t22\t\t\tLoss:\t0.20057132135743883\n",
      "Accuracy:\t0.959875\n",
      "iteration:\t23\t\t\tLoss:\t0.19494548730535913\n",
      "Accuracy:\t0.962\n",
      "iteration:\t24\t\t\tLoss:\t0.18965198246710332\n",
      "Accuracy:\t0.964125\n",
      "iteration:\t25\t\t\tLoss:\t0.18466189353092477\n",
      "Accuracy:\t0.964875\n",
      "iteration:\t26\t\t\tLoss:\t0.17994964653094456\n",
      "Accuracy:\t0.966375\n",
      "iteration:\t27\t\t\tLoss:\t0.17549271294489835\n",
      "Accuracy:\t0.968\n",
      "iteration:\t28\t\t\tLoss:\t0.1712711266593919\n",
      "Accuracy:\t0.96925\n",
      "iteration:\t29\t\t\tLoss:\t0.16726706170854783\n",
      "Accuracy:\t0.970375\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XucXHV9//HXZ647e79vsru5kGRDLhAChPstBUS0VdpqFRSFesFqqdpWfz/sz1pLL1otWvuQ2iJSqUUpakVUNKASQEAgCQFyIfdssrnsfbOX2Z3Zmfn8/jhnk8lmk0z2NjtnPs/HYx7nMmdmPt+c5D3ffM+Zc0RVMcYY4y2+bBdgjDFm8lm4G2OMB1m4G2OMB1m4G2OMB1m4G2OMB1m4G2OMB1m4G2OMB1m4G88Tkb0icn226zBmOlm4G2OMB1m4m7wlIh8WkZ0i0iUij4lIvbteROSrItImIkdE5DUROcd97q0iskVE+kTkgIh8KrutMGZsFu4mL4nItcAXgHcBs4Fm4GH36RuAq4HFQDnwbqDTfe5bwEdUtQQ4B/j1NJZtTMYC2S7AmCx5L/CAqm4AEJHPAN0iMh8YBkqAJcBLqro17XXDwDIReVVVu4Huaa3amAxZz93kq3qc3joAqtqP0ztvUNVfA18H7gVaReQ+ESl1N30H8FagWUSeFpHLprluYzJi4W7y1UFg3siCiBQBVcABAFX9V1W9EFiOMzzzaXf9y6p6E1ALPAo8Ms11G5MRC3eTL4IiUjDywAnlPxaRlSISBv4ReFFV94rIRSJyiYgEgQFgCEiKSEhE3isiZao6DPQCyay1yJhTsHA3+eJxYDDtcRXw18APgUPAQuBmd9tS4Js44+nNOMM1/+w+9z5gr4j0An8C3DpN9RtzRsRu1mGMMd5jPXdjjPEgC3djjPEgC3djjPEgC3djjPGgrP1Ctbq6WufPn5+tjzfGmJy0fv36DlWtOd12WQv3+fPns27dumx9vDHG5CQRaT79VjYsY4wxnpST4Z5IprJdgjHGzGg5F+7/9cJeLv3Cr4kl7FffxhhzMjkX7o0VETr6Y7ywq/P0GxtjTJ7KuXC/fGE1RSE/aza3ZrsUY4yZsXIu3AuCflafXcuTW1pJpuy6OMYYM5acC3eAG5bX0dEfY+N+uwmOMcaMJSfD/XeW1BL0iw3NGGPMSWQU7iJyo4hsc+8Uf9cYz88VkadE5BX3TvFvnfxSjyktCHL5wmrWbD6MXbLYGGNOdNpwFxE/zr0k3wIsA24RkWWjNvss8Iiqno9zw4N/m+xCR7theR3NnVG2tfZN9UcZY0zOyaTnfjGwU1V3q2oceBi4adQ2inP3GoAynPtTTqk3LatDBJ6woRljjDlBJuHeAOxPW25x16X7PHCriLTg3M7sz8Z6IxG5Q0TWici69vb2cZR7TG1JARfMrWDN5sMTeh9jjPGiTMJdxlg3eqD7FuDbqtoIvBX4joic8N6qep+qrlLVVTU1p72o2Wm9eXkdmw/2sr8rOuH3MsYYL8kk3FuAOWnLjZw47PJBnLvJo6ovAAVA9WQUeCo3LJsFwBNbbGjGGGPSZRLuLwNNInKWiIRwDpg+NmqbfcB1ACKyFCfcJzbukoH51UWcXVfCEzY0Y4wxxzltuKtqArgTWANsxTkrZrOI3C0ib3c3+0vgwyLyKvA94HadpnMU37y8jpf3dtHZH5uOjzPGmJyQ0c06VPVxnAOl6es+lza/BbhickvLzA3LZ/Gvv97Jr7a28a6L5pz+BcYYkwdy8heq6ZbXl9JQHrGzZowxJk3Oh7uIcMPyOp7d2cFALJHtcowxZkbI+XAHePPyWcQTKZ7ePuXHcI0xJid4ItxXzaugsihkQzPGGOPyRLgH/D6uW1LLr99oI56w+6saY4wnwh2coZm+oQS/3W233zPGGM+E+5VN1RSG/DY0Y4wxeCjcndvv1fDkllZSdvs9Y0ye80y4gzM009YX45X9PdkuxRhjsspT4b767FoCPuGJLTY0Y4zJb54K97JIkMsWVvHE5la7/Z4xJq95KtzBGZrZ0zHAjrb+bJdijDFZ47lwv2FZHQBrNtnQjDEmf3ku3GtLCzh/brndwMMYk9c8F+7gDM28fuAIB3oGs12KMcZkhWfDHbA7NBlj8lZG4S4iN4rINhHZKSJ3nWSbd4nIFhHZLCLfndwyz8xZ1UUsriu2X6saY/LWacNdRPzAvcBbgGXALSKybNQ2TcBngCtUdTnwySmo9YzcsGwWL+3ponsgnu1SjDFm2mXSc78Y2Kmqu1U1DjwM3DRqmw8D96pqN4Cqtk1umWfuzctnkVL4hfXejTF5KJNwbwD2py23uOvSLQYWi8hzIvJbEblxsgocr3MaSlleX8rf/3QLr7ccyXY5xhgzrTIJdxlj3eiffwaAJmA1cAtwv4iUn/BGIneIyDoRWdfePrV3TRIRvnXbRZQXhrj9P19id7v9qMkYkz8yCfcWYE7aciNwcIxtfqyqw6q6B9iGE/bHUdX7VHWVqq6qqakZb80Zm1VWwHc+eDEA7/vWSxw+MjTln2mMMTNBJuH+MtAkImeJSAi4GXhs1DaPAr8DICLVOMM0uyez0PFaUFPMt//4Yo4MDvP+B16kJ2oHWI0x3nfacFfVBHAnsAbYCjyiqptF5G4Rebu72RqgU0S2AE8Bn1bVGXNLpHMby7jv/ReytyPKB779MtF4ItslGWPMlJJsXT1x1apVum7dumn9zF9sOsTHHtrA1Ytr+Ob7VxH0e/I3XMYYDxOR9aq66nTb5VW63XjObP7hD85l7bZ2PvX9V+2OTcYYzwpku4DpdsvFc+kaiPPlNduoKAzxN29bhshYJwQZY0zuyrtwB/jY6oV09sd54Lk9VBeHuPPaE07sMcaYnJaX4S4ifPZ3l9IdjfPPT2ynoijEey+Zl+2yjDFm0uRluAP4fMKX3rmCnmiczz66iYrCEG89d3a2yzLGmEmRVwdURwv6ffzbey/kgrkV3PndDXzmf1+noz+W7bKMMWbC8jrcASIhP9/+44u4/fKz+P66/az+8lr+/eldxBLJbJdmjDHjlvfhDlBSEORzb1vGmj+/mksXVPLFn7/Bm77yDD9//RDZ+h2AMcZMhIV7moU1xdx/20V854MXEwn6+ehDG3j3fb9l0wG7qqQxJrdYuI/hqqYafvbxK/mHPziHnW39vO3rv+HT33+Vtl678JgxJjdYuJ9EwO/jvZfMY+2nV3PHVQt4dOMBVv/zWr7+6x0Mxm083hgzs+XVtWUmYm/HAF/4+VbWbG6lvDDIzRfN5X2XzaOhPJLt0owxeSTTa8tYuJ+h9c1d3P/sHtZsPoyI8Oblddx++VlcNL/CLmNgjJlymYZ73v6IabwunFfJhfMqOdAzyH+9sJeHX9rP468fZnl9KbdfPp+3nVdPQdCf7TKNMXnOeu4TFI0nePSVg3z7+T1sb+2nqijEey6Zy62XzqOutCDb5RljPMaGZaaZqvL8rk7+87k9/OqNNvwi3HjOLN5xQSNXNlXbteONMZPChmWmmYhwxaJqrlhUTXPnAA8+38wPN7Tw09cOUVkU4nfPnc1NK+u5YG4FPp+NzRtjplZGPXcRuRH4GuAH7lfVL55ku3cC3wcuUtVTdsu91nMfSzyR4unt7fx44wF+ubWVoeEUDeURblpZz00rGzh7Vkm2SzTG5JhJG5YRET+wHXgT0IJzw+xbVHXLqO1KgJ8BIeBOC/fj9ccSPLH5MD/eeJDf7OwgmVKWzCrhppUNvH1lvZ1SaYzJyGSG+2XA51X1ze7yZwBU9QujtvsX4JfAp4BPWbifXHtfjMdfP8SjGw/wyr4eAM6bU871S2q5bmkdS2eX2GmVxpgxTeaYewOwP225Bbhk1IedD8xR1Z+KyKfOqNI8VFMS5rbL53Pb5fPZ1xnlsVcP8OTWNu55cjv3PLmdhvII1y6p5bqltVy6oMpOrTTGnLFMwn2sLuTR7r6I+ICvAref9o1E7gDuAJg7d25mFXrc3KpC7ry2iTuvbaKtb4in3mjjl1vb+MH6Fr7z22YKQ36uaqrmuqV1XLukluricLZLNsbkgAkPy4hIGbAL6HdfMgvoAt5+qqGZfB6WycTQcJIXdnfyq62t/GprG4eODCECKxrLuXJRFVcsrOaCeRXWqzcmz0zmmHsA54DqdcABnAOq71HVzSfZfi025j6pVJUth3r51dY2nt7ezsb9PSRTSijgY9W8Cq5YVM3lC6s4t6GMgJ1Pb4ynTdqYu6omROROYA3OqZAPqOpmEbkbWKeqj028XHMqIsLy+jKW15fx8eua6I8leGlPJ8/t7OS5nR18ec02AErCAS5ZUMnlC53z7RfXFduBWWPylP1C1QM6+2O8sNsJ++d3ddDcGQWgvDDIhXMruHB+BavmVbKiscyGcYzJcfYL1TxSVRzm91bU83sr6gFo6Y7y/K5O1u3tYl1zN796ow2AoF84p6GMVfMquHBeJavmV9gBWmM8ynrueaBrIM765m7WNXexfm83r7UcIZ5MATC/qpAL5lVwXmM5KxrLWDq71Hr3xsxgduEwc1KxRJJNB46wbm8365q7eWVfNx39cQACPuHsWSWsaCznvMYyzm0sY3FdiV34zJgZwsLdZExVOXRkiNdaenit5Yj76KF3KAFAOOBjeX0pKxrLWV5fyrL6UhbVFhMOWA/fmOlmY+4mYyJCfXmE+vIIN54zG3ACv7kzyqtHA7+H/3l5P4PDzv1jAz5hUW0xy2Y7Yb90tvOoLAplsynGGJf13E3Gkillb+cAWw72svVQL1sOOdPW3tjRbWaVFrCsvpQls0o4e1YJTbUlLKgpsnF8YyaJ9dzNpPP7hIU1xSysKeZt59UfXd/ZH2ProT62HDrC1kN9bD3UyzPb20mknI6DT2B+VRFNdcUsriuhqa6ExXXFLKguJhSwsXxjpoKFu5mwquIwVzaFubKp+ui6eCLFno4Btrf2saO1j+2t/Wxv6+OXW9tIuqEf8Anzq4tYWFPEwppiFtQUs7CmiAU1xZRFgtlqjjGeYOFupkQo4ONsd2gmXSyRZHe7E/rb3dDf2dbPr7a2He3pA1QXh46FfXUxC2udaUNFxM7cMSYDFu5mWoUD/qMHX9MNJ1Ps74qyq32A3e397GrvZ3f7AGs2t9I1cOyK036f0FgRYV5VEWdVFTKvqoj51c50TkWhDfMY47JwNzNC0O9jgTs0A3XHPdc9EGd3Rz+72gfY1xllb+cAezsHeKW5m75Y4uh2PoH68gjzq4qYU1nI3MpC5lRGmFPhzJcXBu1aOyZvWLibGa+iKMSFRZVcOK/yuPWqStdAnL2dUZo7B9jbGWVvxwDNnQP8YtMhuqPDx21fHA7QWBE5FvzufH15hIaKCKUFNs5vvMPC3eQsEaGqOExVcZgL51Wc8Hx/LMH+rij7u6Ls64rS0j3I/i7ni+DZHe0MDaeO276kIEBDecR5VDjTkeBvLI9QXRzG57Oev8kNFu7Gs4rDgTHH98Hp9Xf0x2npjnKgZ5AD3YMc7BnkQM8gLd2DvLS3i76hxHGvCfqFutICZpcVMLss4k4LmFUWob68gFllBVQX2ReAmRks3E1eEhFqSsLUlIQ5f+6JvX6A3qFhJ/C7ndA/dGSIQ+504/4efrFp6OgF2EYE/UJtiRP0s0oLqC0NM6u0gLqjjzCzygooDNk/PTO17G+YMSdRWhCkdFaQJbNO7PmD0/vvHIhz+MgQB3sGOdw7xMGeIQ4fGaS1N8bWw72s3TbEQDx5wmtLwgHqygqodb9gakvC1JY4XwY1xWFnWlJAaUHADgKbcbFwN2acRITq4jDVxWHOaSg76Xb9sQSHjwzR1jvE4d4hWntjtPYOcfjIEO39MTbs66atN0YskTrhteGA7+j/MEY+q6Y4RHXacrW7XBK2LwJzTEbhLiI3Al/Duc3e/ar6xVHP/wXwISABtAMfUNXmSa7VmJxUHA6wqLaYRbXFJ91GVemLJWjrjdHeF6Otb4j2vpF5Z7q/K8or+7rpHIgz1iWhQgEfNcVhqopDVBWFqCxygr+q2JmvKg5RXRSm0n3ervfjbacNdxHxA/cCbwJagJdF5DFV3ZK22SvAKlWNishHgS8B756Kgo3xIhFxhoEKgqf8EgDnAm5dA3Ha+2J09Kc/4nT0xegciNPeH2Pb4T46BuLEx/gfAUBRyE9FkRP0FUUhKgvdqfuoKByZD1JeGKI8ErQbsOeQTHruFwM7VXU3gIg8DNwEHA13VX0qbfvfArdOZpHGmGP8vmMHg09HVRmIJ+nsd0K/sz9+3HxPNE5XNE7XQJydbf10D8THPEYwoqQgQEVhiIpCN/ALg1SMmo58EZQXBimLBCkpCOK3M4imXSbh3gDsT1tuAS45xfYfBH4+1hMicgdwB8DcuXMzLNEYM14iQnE4QHE4wLyqooxeMzScpCc6TNeAE/pdUedLoHtgmO6R+egwPdE4ezoG6I7GTzht9PganIPT5YVByiNBSiPOF0BZJEBZxPkCKC0IHptPm5aEA3Zq6ThlEu5j/cmOeRF4EbkVWAVcM9bzqnofcB8413PPsEZjzDQqCPqZVeZnVllBxq9JJFP0DDqBf2RwmJ6o8zgyOEzP4DBHonH3eWfd/q4ovUMJjgwOH71K6Fh8AiUFQUoKAs6wVcSZlhw3H6DU/YIoLQgc3b7Enc/X6w1lEu4twJy05Ubg4OiNROR64P8B16hqbPTzxhjvCvh9R8/eORMjw0ZHBofpHXSCf+TRm7aubyhB79AwvYMJ9nVFneXB4eOuLXQy4YDvaNAfDf2wM19cEKAk7EyLw8FRy8ceReFAzn1JZBLuLwNNInIWcAC4GXhP+gYicj7wH8CNqto26VUaYzwpfdiooTxyxq9PppR+N/hHvgT6hpxpf+zYfG/a+r6hYdp6YwzEEvTFnO0yuSFdKOA7LuxLwgGKwn5nviBAUchZP7KuOBygMOQuu88VhwMUustTfRzitOGuqgkRuRNYg3Mq5AOqullE7gbWqepjwJeBYuD77nm2+1T17VNYtzHG4PcJZYVBygqDxw0vnAlVJRpPul8Gx74U+oeOLQ/EEvTHE/QPufOxJP2xYTr64zR3Rulzt4me4mB0ur+7aTnvu2z+OCvOTEbnuavq48Djo9Z9Lm3++kmuyxhjpoWIuD3uAHVj/xg5Y6mUEh1OEo2NfCkkGYiPfCE44T8QS5xwhdOpYL9QNcaYSeLzHRtmqs12LVn+fGOMMVPAwt0YYzxINJPDxFPxwSLtwHivP1MNdExiOTOB19rktfaA99rktfaA99o0VnvmqWrN6V6YtXCfCBFZp6qrsl3HZPJam7zWHvBem7zWHvBemybSHhuWMcYYD7JwN8YYD8rVcL8v2wVMAa+1yWvtAe+1yWvtAe+1adztyckxd2OMMaeWqz13k8dEZK2IdIvImV2lypg8YuFucoqIzAeuwrns9LRdv0hE7NfcJqfkXLiLyI0isk1EdorIXdmuZ6JEZK+IvC4iG0VkXbbrGQ8ReUBE2kRkU9q6ShF5UkR2uNOKSfq49+Pc7evbwG1pnxcRkXtEpFlEjojIb0Qk4j53pYg8LyI9IrJfRG53168VkQ+lvcft7utG2qMi8qcisgNoF5EDItIuInERiYrIehG5Ku31fhH5KxHZJSJ97vNzROReEbln1J/ZT0Tkk5P0Z3Jabh1PichWEdksIp9w10/VfppSp2jP5939tNF9vDXbtWZKRApE5CURedVt09+6688SkRfdffQ/IhLK6A1VNWceOFel3AUsAELAq8CybNc1wTbtBaqzXccE23A1cAGwKW3dl4C73Pm7gH+apM/aCXwMuBAYBurc9fcCa3HuHOYHLgfCwFygD7gFCAJVwEr3NWuBD6W99+3Ab9Lao8CTQCXw98CncG4hWYVzXaa/BA4DBe7rPw28DpyNc5Ob89xtL8a5B4LP3a4aiI7UPk37aDZwgTtfAmwHlk3Vfspiez4PfCrb9Y2zTQIUu/NB4EXgUuAR4GZ3/b8DH83k/XKt5370fq6qGgdG7udqskhVnwG6Rq2+CXjQnX8Q+P2Jfo6IXAnMAx5R1fU4X/TvEREf8AHgE6p6QFWTqvq8OjeNeS/wS1X9nqoOq2qnqm48g/Z8QVW7gIT73H+775FQ1XtwvkDOdrf9EPBZVd2mjlfdbV8CjgDXudvdDKxV1daJ/plkSlUPqeoGd74P2IrzRTjp+2k6nKI9Ocv9O9PvLgbdhwLXAj9w12e8j3It3Me6n2tO71CcnfeE+1/4O7JdzCSqU9VD4PxDhEm5SN5twBOqOvJz7O+666qBApywH23OSdZnKv3v250ictAd3jkiIj1Amfv5p/usBzl24/hbge9MoKYJcY9bnI/TM5yK/TStRrUHnP30mju8lhPDTCPcob2NQBvO/xp3AT2qOnLLqYwzL9fCPeP7ueaQK1T1AuAtwJ+KyNXZLmgmcsfP3wVcIyKHReQw8Oc4Qx+zgSFg4Rgv3X+S9QADQGHa8qwxthn5+/UNnC+SAM5/k3+oquU4PfKRv5en+qz/Bm4SkfOApcCjJ9luSolIMfBD4JOq2puNGibTGO35Bs4+WAkcAu45xctnHPd/nStxbmd6Mc7flRM2y+S9ci3cM7qfay5R1YPutA34Ec4O9YJWEZkN4E4nevvF3weSOOOqK93HUuBZnIOsDwBfEZF6t/dzmXuq5EPA9SLyLhEJiEiViKx033Mj8IciUigii4APnuzD3SGUIpzhmW8Cl4jI54D02zvcD/ydiDSJY4WIVLmvb8G5ZeV3cL4YBif453HGRCSIE4QPqer/uqsnez9Nm7Hao6qtbkCmcPZTTv57UtUenGNClwLlcuxsrYwzL9fC/ej9XN0jxjcDj2W5pnETkSIRKRmZB24ANp36VTnjMY6dzXIb8OMJvt9twH+q6j5VPTzyAL6OM65+F87BzJdxxsv/CecA5j7grTgHP7twAv089z2/CsSBVpxhk4dO9uFu8K0Bfo7zhbIA538L6cM2X8Hp1T8B9ALfAtJvDPogcC5ZGJIREXHr2aqqX0l7arL307Q4WXtGvqhcf0AO/XsSkRoRKXfnI8D1OMcSngLe6W6W8T7KuV+ouqc2/QvH7uf6D1kuadxEZAFObx2c/+5/NxfbIyLfA1bjjD23An+DM+zwCM7ZKvuAP3IPTM54J2nPapz/LSjOGU4fGRmrPoP3vRpneGa+27OcNu7B6GdxvgBHPvuvcMapc24/naI9tzDB/ZQtIrICpwPgx+l4P6Kqd7s58TDOWVuvALe6Jwuc+v1yLdyNyUXuEMLDwKuqene26zHel2vDMsbkHBFZCvTgHPj9lyyXY/KE9dyNMcaDrOdujDEelLWLIVVXV+v8+fOz9fHGGJOT1q9f36EZ3EM1a+E+f/581q3LyetkGWNM1ohIcybb2bCMMcZ4kF2j2hhjxklVSaaUeDLFcMKZJlLH5ofTHvGEOs8lUzTVljCnsvD0HzABFu7GmJyQSimJlJJSZ5pMOmGZdNcnU0oskSKWSBJLpIgnUs7ycJJ4MkVsOHXc885ykqFR01gixdCwu437PvFEkuGkEk+MBHWKeNJ5jOeEw7///XO49dJ5k/+HlMbC3RgzIeqG7UgoDsaTRONJovEEg/Ekg8PO8qC7LnrcNkkG4wlnOpxkIHZsfuQ1sUSSRErHFaKnE/QL4YCfcMBHQdCZhkemAR9lkSDhgI+Q30fQL4QCPoJ+HyF3XfpywCeE3eWg30cw4CPkF4J+H4GR17vPNVZETl/cBFm4G+MxiWSKoUSKwXjSCdvh5NHQdeZTx60fWR5KJImNzI+sTySP9WKHnZ5qzN3uWO84SeoMg9cnUBgKEAn5KQz5iQT9R+cri8IUjqwP+SkI+gn6BJ9PCPgEv8/nToWA3536BJ8I4aCfkN9HOOg7GtAj4R0O+Am560amAb93DztauBszQyRTSv9Qgt6hYY4MDtM7NEzv4DC9g866/liCgViC/liCvqFR8/EE/UMJBmLOEMR4hAI+CtwerPNw5wN+isMBqoqOBeVIOB4N08BIoB4LaSe0A8cFdZEb6OGAD+faX2aqWLgbM8lSKaV3aJju6DDd0TjdA3G6o8P0RON0jZrvHUq4AT5MXyxx2vcuDPkpCgcoCQcoCgcoDgdorCikpMCZLwo7YVoQ9BFxQzoScgJ6pBccSQvukW3CAR8+n4Wtl1i4G3MKqZTSFY3T3hejayBOjxvYPdGRkB52553negad5ZMNU/h9QkVhkIrCEBWFIRrKIyybXUppJEBpQZDSSJDSggBlkZH5IKWRACUFQYrDAfwWwCZDFu4mLyVTSmd/jNbeGG19Q7T1xWjrjdHaN0Rbb4x2d117X4zESZK6MOSnPBKkvDBERVGQ2eURKgqDlEdCVBSFnBAvCrlB7syXhAM2HGGmhYW78ZRkSukaiDuB3RujtXeI1rTQbusborV3iPa+2Ji968qiELUlYWpKwiyqLaGuNExtSZja0gIqi0KUu73uskiQgqB/+htoTIYs3M2Mp6p09Mc52DNI50CMjr447f0xOvvjdPTH6Eib74rGxzxlbiS060oLWDKrhLrSAmpLC5zgdsO7pjhMKODdsydMfrFwN1mn6vS2W7oHaekeZH93lJbuqDPf5UxjiRPPACkK+akqDlNdHGJeVSEXzKugpjhEVfGxwK4rdXrh4YD1sk1+sXA3Uy6VUtr7Y7R0D3KgZ5AD3YMc6Im6UyfQo/Hkca8pLwzSWBGhqbaEa5fU0lhRyOyyAmpKwlQXO49IyALbmJOxcDeTIpFMsa8rys62fna297OnfcAJ8p5BDvUMnXDudXlhkIbyCPOrirhyUQ2NFRHmVBbSWBGhoSJCaUEwSy0xxhss3M0ZGRpOsqdjwAnxtMeejoHjAry2JExjRYQVjeW85RwnsBvLnWl9eYTisP3VM2Yq2b8wMyZVpbU3xtZDvWw51MuWg860uXPg6FkmPoE5lYU01RazekkNi2qKaaorYWFNESXW8zYmqyzcDYlkit0dA0cDfGTaNRA/us3cykKWzi7hbefVs6i2mKbaYs6qLrLTAY2ZoSzc84yqsq8rysb9Pbyyr4dXW3rYfLCXuHvBQKqPAAANmklEQVQ2Sijg4+y6Et60tI5l9aUsqy9lyawS64kbk2Ms3D3uSHSYjS09bNzXw8b93bzacuRojzwS9HNuQxm3XTaP5fVlLKsvZUF1kaevlGdMvrBw95iO/hjP7ezg2R0dbGjuZnfHAAAisKimmOuW1LJybjkr55Rzdl2JBbkxHmXhnuPiiRTrm7t5Zkc7z2xvZ/PBXsA51XDVvErecWEjK+eUs6KxzIZWjMkjFu45RlXZ0zHAM9vbeXZHBy/s7iQaTxLwCRfMreBTNyzmqqYazmkosysIGpPHLNxzQN/QMM/t7DzaO2/pHgRgXlUh77igkauaqrlsYZX1zI0xR1m4z0CplLLlUC9Pb2/n6e3tbGjuJpFSisMBLltYxUeuWcjVTdXMqyrKdqnGmBnKwn2G6OyP8eyODp7e3s6zO9rp6HfOaFleX8odVy/gmsU1XDCvgqAdADXGZMDCPYsO9Azys9cO8rPXD/NaSw+qzqVpr2qq5prFNVzVVENNSTjbZRpjcpCF+zRr6x3iZ68f4qevHWJ9czcA5zaU8RfXL+aas2s4p77M7mVpjJkwC/dp0Nkf4+ebDvPT1w7y4p4uVGHJrBI+/eaz+d1zZzO/2sbOjTGTy8J9igzGk/zk1YP85LWDPL+rk2RKWVhTxMevbeJt581mUW1Jtks0xniYhfsU+M2ODj7zo9fY3zXI3MpCPnL1An5vRT1LZ5fYzZGNMdPCwn0S9UTj/P3PtvKD9S0sqC7ioQ9dwuULqyzQjTHTzsJ9Eqgqj79+mL95bDPd0TgfW72Qj1/XZJfDNcZkjYX7BB0+MsRf/3gTT25p5ZyGUh78wEUsry/LdlnGmDxn4T5OqZTy8Mv7+cLjW4knU3zmLUv44JVn2VUWjTEzgoX7OOzpGOCuH77Gi3u6uGxBFV/4w3PtdEZjzIySUbiLyI3A1wA/cL+qfnHU83OBB4Fyd5u7VPXxSa4164aGk3zrN3v42q92EA74+OIfnsu7L5pjB0yNMTPOacNdRPzAvcCbgBbgZRF5TFW3pG32WeARVf2GiCwDHgfmT0G9WZFKKY9uPMA9T2znQM8gNy6fxd03Lae2tCDbpRljzJgy6blfDOxU1d0AIvIwcBOQHu4KlLrzZcDBySwym57Z3s4Xfv4GWw/1cm5DGV9+5wouX1Sd7bKMMeaUMgn3BmB/2nILcMmobT4PPCEifwYUAddPSnVZtOnAEf7pF2/w7I4O5lRG+NrNK3nbinq77osxJidkEu5jpZmOWr4F+Laq3iMilwHfEZFzVDV13BuJ3AHcATB37tzx1DvlWrqj3PPEdn70ygHKC4P89e8t49ZL5xIO2DnrxpjckUm4twBz0pYbOXHY5YPAjQCq+oKIFADVQFv6Rqp6H3AfwKpVq0Z/QWRVTzTOvU/t5MHnmxGBj65eyJ9cs5CyiN3dyBiTezIJ95eBJhE5CzgA3Ay8Z9Q2+4DrgG+LyFKgAGifzEKn0lPb2vjE916hL5bgnRc08hc3LGZ2WSTbZRljzLidNtxVNSEidwJrcE5zfEBVN4vI3cA6VX0M+EvgmyLy5zhDNrer6ozqmZ/MrvZ+Pv7dV2isLOSr7z6PJbNKT/8iY4yZ4TI6z909Z/3xUes+lza/Bbhickubev2xBH/ynfUEAz7uv20VDeXWWzfGeEPe/lZeVfk/P3iVXe39fP2W8y3YjTGekrfh/h/P7Obx1w9z11uW2HnrxhjPyctw/82ODr70izf43RWz+fBVC7JdjjHGTLq8C/f9XVH+7HsbWFRbzJfescKuC2OM8aS8Cveh4SQffWg9iZTyH+9bRVHYLoppjPGmvEk3VeX//WgTmw708q3bVnGWXaLXGONhedNz/+/fNvPDDS184romrltal+1yjDFmSuVFuK9v7uJvf7KFa5fU8onrmrJdjjHGTDnPh3tb7xAf/e8NNFRE+Oq7V9pVHY0xecHT4R5PpPjYQxvoG0rwH++70C4CZozJG54+oPqPj29lXXM3/3rL+XbNGGNMXvFsz717IM6DL+zlvZfM5e3n1We7HGOMmVaeDfdndrSjCu+8sDHbpRhjzLTzbLg/vb2disIgKxrLs12KMcZMO0+GeyqlPLO9nauaavDb2THGmDzkyXDfcqiXjv44q8+uyXYpxhiTFZ4M97XbnFu3XtVk4W6MyU+eDPent7dzbkMZNSXhbJdijDFZ4blwPzI4zIZ9PVyz2Hrtxpj85blwf25nB8mU2ni7MSaveS7c125ro7QgwMo5dgqkMSZ/eSrcVZWn3VMgA35PNc0YY86IpxLwjcN9tPbGuMaGZIwxec5T4f709nYAO5hqjMl7ngr3tdvaWDq7lLrSgmyXYowxWeWZcO+PJVi3t9t67cYYg4fC/bmdHSTsFEhjjAE8FO5rt7VTHA5w4byKbJdijDFZ54lwV3WuAnnFoiqCdgqkMcZ4I9x3tvVzoGeQaxbXZrsUY4yZETwR7kdPgbTxdmOMATwS7mu3tdNUW0xDeSTbpRhjzIyQ8+EejSd4aU+XnSVjjDFpcj7cX9jVSTyZsvF2Y4xJk1G4i8iNIrJNRHaKyF0n2eZdIrJFRDaLyHcnt8yTe3p7O5Ggn4vOslMgjTFmROB0G4iIH7gXeBPQArwsIo+p6pa0bZqAzwBXqGq3iExLN1pVWbutncsXVhEO+KfjI40xJidk0nO/GNipqrtVNQ48DNw0apsPA/eqajeAqrZNbplj29sZZV9X1MbbjTFmlEzCvQHYn7bc4q5LtxhYLCLPichvReTGsd5IRO4QkXUisq69vX18FacZuRG2jbcbY8zxMgl3GWOdjloOAE3AauAW4H4ROeFWSKp6n6quUtVVNTUT720/vb2dBdVFzK0qnPB7GWOMl2QS7i3AnLTlRuDgGNv8WFWHVXUPsA0n7KfM0HCSF3Z1crVdBdIYY06QSbi/DDSJyFkiEgJuBh4btc2jwO8AiEg1zjDN7sksdLQX93QRS6RsvN0YY8Zw2nBX1QRwJ7AG2Ao8oqqbReRuEXm7u9kaoFNEtgBPAZ9W1c6pKhqc8fZwwMelC6qm8mOMMSYnnfZUSABVfRx4fNS6z6XNK/AX7mNaPL29nUsXVFEQtFMgjTFmtJz8her+rii72wfsrkvGGHMSORnua92rQNp4uzHGjC0nw/3pbW3MqYxwVnVRtksxxpgZKefCPZZI8vyuTlYvrkVkrFPwjTHG5Fy4r9vbTTSetPF2Y4w5hZwL9w3N3YT8Pi5fZKdAGmPMyWR0KuRMcue1i/ijVXMoDOVc6cYYM21yrucuIswqK8h2GcYYM6PlXLgbY4w5PQt3Y4zxIHGuHJCFDxZpB5rH+fJqoGMSy5kJvNYmr7UHvNcmr7UHvNemsdozT1VPe7pg1sJ9IkRknaquynYdk8lrbfJae8B7bfJae8B7bZpIe2xYxhhjPMjC3RhjPChXw/2+bBcwBbzWJq+1B7zXJq+1B7zXpnG3JyfH3I0xxpxarvbcjTHGnIKFuzHGeFDOhbuI3Cgi20Rkp4jcle16JkpE9orI6yKyUUTWZbue8RCRB0SkTUQ2pa2rFJEnRWSHO63IZo1n4iTt+byIHHD300YReWs2azxTIjJHRJ4Ska0isllEPuGuz8n9dIr25Ox+EpECEXlJRF512/S37vqzRORFdx/9j4iEMnq/XBpzFxE/sB14E9ACvAzcoqpbslrYBIjIXmCVqubsDy9E5GqgH/gvVT3HXfcloEtVv+h+CVeo6v/NZp2ZOkl7Pg/0q+o/Z7O28RKR2cBsVd0gIiXAeuD3gdvJwf10iva8ixzdT+LcoKJIVftFJAj8BvgEzr2p/1dVHxaRfwdeVdVvnO79cq3nfjGwU1V3q2oceBi4Kcs15T1VfQboGrX6JuBBd/5BnH94OeEk7clpqnpIVTe4833AVqCBHN1Pp2hPzlJHv7sYdB8KXAv8wF2f8T7KtXBvAPanLbeQ4zsUZ+c9ISLrReSObBcziepU9RA4/xCB2izXMxnuFJHX3GGbnBi+GIuIzAfOB17EA/tpVHsgh/eTiPhFZCPQBjwJ7AJ6VDXhbpJx5uVauI91X73cGVca2xWqegHwFuBP3SEBM/N8A1gIrAQOAfdkt5zxEZFi4IfAJ1W1N9v1TNQY7cnp/aSqSVVdCTTijFQsHWuzTN4r18K9BZiTttwIHMxSLZNCVQ+60zbgRzg71Ata3XHRkfHRtizXMyGq2ur+w0sB3yQH95M7jvtD4CFV/V93dc7up7Ha44X9BKCqPcBa4FKgXERG7k6UceblWri/DDS5R49DwM3AY1muadxEpMg9GISIFAE3AJtO/aqc8Rhwmzt/G/DjLNYyYSMB6PoDcmw/uQfrvgVsVdWvpD2Vk/vpZO3J5f0kIjUiUu7OR4DrcY4lPAW8090s432UU2fLALinNv0L4AceUNV/yHJJ4yYiC3B66+Dc8vC7udgeEfkesBrn8qStwN8AjwKPAHOBfcAfqWpOHKQ8SXtW4/xXX4G9wEdGxqpzgYhcCTwLvA6k3NV/hTNOnXP76RTtuYUc3U8isgLngKkfp+P9iKre7ebEw0Al8Apwq6rGTvt+uRbuxhhjTi/XhmWMMcZkwMLdGGM8yMLdGGM8yMLdGGM8yMLdGGM8yMLdGGM8yMLdGGM86P8DPoIuqH14U0wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time for GD is:\t438.55300283432007\n"
     ]
    }
   ],
   "source": [
    "# GD 算法\n",
    "import time\n",
    "input_dim = 784\n",
    "num_classes = 2\n",
    "mini_batch = 8000\n",
    "model3 = LogisticRegression(input_dim, num_classes, lr=1000, batch=mini_batch, weight_scale=4e-2, weight_decay=1.0, reg=0.0)\n",
    "\n",
    "tstart = time.time()\n",
    "model3.train(new_train_data[0:mini_batch], new_train_labels[0:mini_batch])\n",
    "tend = time.time()\n",
    "\n",
    "# 画图\n",
    "Loss, Acc = model3.history()\n",
    "plt.figure\n",
    "plt.subplot(2,1,1)\n",
    "plt.plot(Loss)\n",
    "plt.title('Loss')\n",
    "plt.subplot(2,1,2)\n",
    "plt.plot(Acc)\n",
    "plt.title('Accuracy')\n",
    "plt.show()\n",
    "print('Time for GD is:\\t%s' % (tend - tstart))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iteration:\t0\t\t\tLoss:\t0.9821093134674342\n",
      "Accuracy:\t0.461\n",
      "iteration:\t1\t\t\tLoss:\t3.8137505820324087\n",
      "Accuracy:\t0.703\n",
      "iteration:\t2\t\t\tLoss:\t1.2876530014756593\n",
      "Accuracy:\t0.716\n",
      "iteration:\t3\t\t\tLoss:\t1.5024046212279936\n",
      "Accuracy:\t0.772\n",
      "iteration:\t4\t\t\tLoss:\t0.9693024779768684\n",
      "Accuracy:\t0.717\n",
      "iteration:\t5\t\t\tLoss:\t1.6867045552417912\n",
      "Accuracy:\t0.779\n",
      "iteration:\t6\t\t\tLoss:\t0.6030759563885674\n",
      "Accuracy:\t0.811\n",
      "iteration:\t7\t\t\tLoss:\t0.757866438564851\n",
      "Accuracy:\t0.861\n",
      "iteration:\t8\t\t\tLoss:\t0.5815392097473006\n",
      "Accuracy:\t0.808\n",
      "iteration:\t9\t\t\tLoss:\t0.8038641692243766\n",
      "Accuracy:\t0.862\n",
      "iteration:\t10\t\t\tLoss:\t0.24689776625805343\n",
      "Accuracy:\t0.916\n",
      "iteration:\t11\t\t\tLoss:\t0.12212163861476187\n",
      "Accuracy:\t0.968\n",
      "iteration:\t12\t\t\tLoss:\t0.07354394827061304\n",
      "Accuracy:\t0.978\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VPW9//HXJ5N9D2QhGwn7TiBEFJeKOwKCglutW+vSe7torV20ve3t9ru3q7W9ett6rXWtFgmIWhRXaqugkATCDgEDmeyE7PvMfH9/zIARWZIwkzMz+Twfj3kkc+bMOZ9Dwjvf+Z7v+R4xxqCUUiq4hFhdgFJKKe/TcFdKqSCk4a6UUkFIw10ppYKQhrtSSgUhDXellApCGu5KKRWENNxV0BORchG51Oo6lBpKGu5KKRWENNzVsCUid4lImYgcEZGXRSTDs1xE5LciUicizSJSKiLTPa8tFJGdItIqIpUi8i1rj0KpE9NwV8OSiFwM/DdwPZAOHARe8Lx8OfA5YCKQCNwANHhe+zPwZWNMHDAdeGcIy1aq30KtLkApi3wBeMIYUwwgIg8CjSKSC/QCccBk4CNjzK4+7+sFporIVmNMI9A4pFUr1U/aclfDVQbu1joAxpg23K3zTGPMO8AjwKNArYg8JiLxnlWXAwuBgyLyDxGZN8R1K9UvGu5quKoCco4+EZEYYCRQCWCM+b0xZg4wDXf3zLc9yzcZY5YCqcBLwIohrlupftFwV8NFmIhEHn3gDuUvisgsEYkA/gv40BhTLiJnicjZIhIGtANdgFNEwkXkCyKSYIzpBVoAp2VHpNQpaLir4WIt0NnncQHwA6AQqAbGATd61o0H/g93f/pB3N01v/a8dgtQLiItwL8BNw9R/UoNiOjNOpRSKvhoy10ppYKQV8NdRGwiUiIir3pzu0oppQbG2y33e4Fdp11LKaWUT3kt3EUkC1gEPO6tbSqllBocb16h+jDwHdxX9p2QiNwN3A0QExMzZ/LkyV7cvVJKBb+ioqLDxpiU063nlXAXkcVAnTGmSETmn2w9Y8xjwGMABQUFZvPmzd7YvVJKDRsicvD0a3mvW+Y8YImIlOOefOliEXnWS9tWSik1QF4Jd2PMg8aYLGNMLu4LQd4xxujFHUopZREd534CrV29LP/DB3yw/7DVpSil1KB4PdyNMeuNMYu9vd2h9PfSaooONvLIO2VWl6KUUoOiLfcTKCy2A/DB/gbK6lotrkYppQZOw/045Yfb2VTeyJ3njyHcFsIzG/p1YloppfyKhvtxVhXbEYE7LhjDopnpFBZX0tbtsLospZQaEA33PlwuQ2FxJeePTyY9IYpb5uXQ1u1gdUml1aUppdSAaLj3sfHjBiqbOrl2ThYAs7MTmZ4ZzzMbytGpkZVSgUTDvY/CokriIkK5fOooAESEW8/JZW9tGx9+fMTi6pRSqv803D3aux28tr2aRTPTiQq3HVt+VV4GCVFhemJVKRVQNNw9XtteQ0eP81iXzFFR4TauL8hi3Y4aalu6LKpOKaUGRsPdY2VRBbkjo5mTk/SZ124+JwenMfz1w0MWVKaUUgOn4Q5UHOlg44EjLM/PQkQ+83rOyBgunJjC8x8dotfpsqBCpZQaGA13YFWxe6jjNfmZJ13n1nk51LV2s25HzVCVpZRSgzbsw90YQ2GxnXljR5KVFH3S9S6cmEr2iCie1hOrSqkAMOzDfVN5I4eOdHzmROrxbCHCzWfn8NHHR9hd0zJE1Sml1OAM+3AvLLITHW5jwfRRp133+oJsIkJ1vhmllP8b1uHe2ePk79uqWTgjnZiI099xMCkmnKvyMlhdUklLV+8QVKiUUoMzrMN93Y4a2rodLM8/dZdMX7fOy6Gjx8mqIrsPK1NKqTMzrMO9sNhOVlIUZ48Z0e/3zMxKJC87kWc2HtT5ZpRSfmvYhntVUyf/KjvMsvwsQkI+O7b9VG49J4f99e18sL/BR9UppdSZGbbhvrqkEmNg+SnGtp/MopnpjIgJ5+kN5V6vSymlvGFYhrsxhsIiO3NzR5AzMmbA748Ms3F9QTZv7qylqqnTBxUqpdSZGZbhXlLRxIHD7SyfM/BW+1FfOHs0BnS+GaWUXxqW4b6yyE5kWAgLZ6QPehvZI6K5ZHIqL2w6RLfD6cXqlFLqzA27cO/qdfLq1ioWTBtFXGTYGW3rlnm5HG7r4fXtOt+MUsq/DLtwf2tXLS1dDq6dk33G27pgfDK5I6N1vhmllN8ZduG+sshOekIk88aNPONthYQIN5+TQ9HBRrZXNnuhOqWU8o5hFe51LV28t7eeZfmZ2AY4tv1krpuTTWSYzjejlPIvwyrcV5dU4jKwbADTDZxOQnQYV8/KZM3WSpo7dL4ZpZR/GDbhfnTe9vzRiYxLifXqtm+Zl0NXr4sXiyq8ul2llBqsYRPu2yqb2VvbxvLTzNs+GNMyEpiTk8QzGw/icul8M0op6w2bcC8sshMeGsLimRk+2f6t83I42NDBe/vqfbJ9pZQaiGER7t0OJ2u2VnH51DQSos5sbPvJLJg+iuTYcD2xqpTyC8Mi3N/dXUdTR69PumSOigi1ceNZo3lnTx0VRzp8th+llOqPYRHuK4vspMZFcMH4ZJ/u56azRyPAsx9q610pZa2gD/fDbd2s31PPNbMzCbX59nAzEqO4bGoaKzZV0NWr880opawT9OG+ZksVDpfxaZdMX7fOy6Wxo5dXS6uHZH9KKXUiQR/uK4vszMxKYGJa3JDs79xxIxmXEsMzG8qHZH9KKXUiQR3uO6qa2VXdwrVD1GoHEBFuOSeHrfZmtlY0Ddl+lVKqr6AO98KiSsJswlU+Gtt+MsvmZBEdbtPZIpVSlgnacO91ulizpZJLJqeRFBM+pPuOjwzjmtmZvFJaxZH2niHdt1JKgZfCXUSyReRdEdklIjtE5F5vbPdMrN9TT0N7z5B2yfR167xcehwuVmzW+WaUUkPPWy13B3C/MWYKcA7wVRGZ6qVtD0phkZ2RMeFcOCnFkv1PGhXH3DEjeHbjQZw634xSaoh5JdyNMdXGmGLP963ALmDwd58+Q43tPby9u5arZ2cS5uOx7ady67wc7I2drN9TZ1kNSqnhyevJJyK5wGzgwxO8dreIbBaRzfX1vptg6+WtVfQ6Dcu9OG/7YFwxbRSpcRF6YlUpNeS8Gu4iEgsUAt8wxrQc/7ox5jFjTIExpiAlxXfdJYXFdqamxzM1I95n++iPMFsIn587mn/sraf8cLultSilhhevhbuIhOEO9ueMMau8td2B2lvbSqm9eciuSD2dm84eTWiI8OxGbb0rpYaOt0bLCPBnYJcx5iFvbHOwCovshIYIS2cN7dj2k0mLj+SKaaNYsbmCzh6db0YpNTS81XI/D7gFuFhEtngeC7207X5zOF2sKqlk/qRUkmMjhnr3J3XLvBxauhy8vLXS6lKUUsNEqDc2Yoz5FyDe2NaZ+GfZYepbuy0b234yZ48ZwcS0WJ7ecJDrC7Jxf9BRSinfCaorVFcW2UmKDuPiyalWl/IpIsIt83LZUdVC8SGdb0Yp5XtBE+7NHb28ubOWJXkZhIf632FdMzuT2IhQnS1ykLp6nfxq3W4eXFVKj8NldTlK+T2vdMv4g1dKq+hxuLh2TrbVpZxQbEQoy/Mzef6jCv5jcbdfnRPwd6X2Ju5fsZV9dW0AHGnv4ZGb8i29QE0pfxc0/zsKi+1MTItleqa1Y9tP5ZZ5OfQ4Xfxtk8430x89DhcPvbmXa/73A1q7HDz1pbn8cPFU1u2o5b6/bcHh1Ba8UicTFC33/fVtlBxq4nsLJ/v1ycrxqXGcO24kz208yJc/N9bnt/0LZLtrWrh/xVZ2VLWwLD+T/7xqGglRYVw4MYVep4v/fm03YbYQfn1dHrYQ//2ZK2WVoEiXwiI7IQJXz7JsOpt+u3VeDlXNXby92/fzzXQ7nGytaOKtnbV0OwJjjL3TZfjD+v0s+Z/3qW3p4k+3zOGh62eREBV2bJ0vXziO+y+byOqSSh5cVYpLJ2ZT6jMCvuXudBlWl1Ry4cQUUuMjrS7ntC6dkkZ6QiTPbDjIFdNGeW27vU4Xe2pa2VbZTKm9mW2VTeypaaXX6Q6+zMQovnrReK6dk+WXJ5wBDtS38a0Xt1J8qIkrp4/iZ1dPZ+RJzk18/ZIJ9Dpd/P6dMsJsIfzs6ul+/alNqaEW8OH+wf7DVDd38f1FU6wupV9CbSHcNHc0v3lzL/vr2xiXEjvgbTicLsrq29whbm+mtNJ9O8Gjo0gSosKYmZXAXReMZWZWAraQEB59t4zvrd7G/64v4+sXj2dZfpbfnJB0uQxPbyjn56/vJiLUxu9unMWSvIzThvV9l02k2+niT/84QJgthP+8aqoGvFIeAR/uhUV24iNDuXRKmtWl9NuNc0fz+3f28cyGg/xoybRTrutyGQ4cbmdbZROldnerfEdVM1297iCPjQhlemY8t5+by8ysBGZmJpI9IuozIXfplFTW76nnt2/t5buF23j03f187eLxLJudaWnff8WRDr6zspQNBxq4aFIKP18+k7R+fgITER5YMJleh+GJ9z8mPDSEB6/07/MuSg2VgA731q5eXt9Rw/L8LCLDbFaX028pcRFcOT2dwiI7375iEjER7h+DMYaDDR2UVjazze4O8+2VzbR75qSJCrMxPTOem+bmMDMrgRlZCYwZGUNIP04oiggXTU5l/qQU3tldx2/f2st3Vpby6Ltl3HPxBJbOyhjSkDfG8LdNFfz01Z2ICL9YPmNQV++KCD9YPIVep4vH3jtAuC2Eb10xyUdVKxU4Ajrc126rpqvX5TczQA7ErfNyeHlrFb94fTfR4aFsq2xim72Zli4HAOGhIUxNj+faOVnMyEpkZlYC41Jiz3hkiIhwyZQ0Lp6cyps7a3n4rX3c/+JWHnm3jHsuGc+SvEyfjz6pbeniu4WlrN9Tz7yxI/nVdTPJSooe9PZEhB8vmUav08Uj75YRHhrCPZdM8GLFSgWegA73lUV2xqbEMDs70epSBmxOThLTMuJ5esNBwmzC5FHxLM7LYGamu0U+MS3Op33iIsLl00Zx6ZQ03thZy8Nv7eW+v23lkXfKuOeSCSyemeH1kDfG8PLWKn64ZgfdDic/XjKNW87J6dcnj9MJCRH+65oZ9DjdY+PDbCH8+/xxXqhaqcAUsOFefridTeWNfPuKSQHZxyoi/OWLZ1HT3MWkUXFEhFrTrRQSIiyYPorLp6bx+o4aHn5rL/e+sIX/eaeMb1w6gYXT070Svg1t3fzHS9t5bXsN+aMT+c31sxiTHOOFI/hESIjwq2vz6HUafvH6bsJswp0XjPXqPpQKFAEb7quK7YjAsnz/H9t+MqlxkaTG+cfwzZAQYeGMdBZMG8Xa7dX87q19fO2vJUxKK+PeSyewYNqoQYf8uh01fG/VNlq7HDxw5WTuumCsz7p+bCHCQ9fn4XC6+NnfdxEeGsKt83J9si+l/FlAhrvLZSgsruT88cmkJ0RZXU5QCQkRFs/M4Mrp6bxaWsXv3t7HV54rZvKoOL5x6USumJbW709KzR29/PiVHawqqWRaRjx/vWsWk0bF+fgI3Lc3/N2Ns+l9rogfrtlx7HaHSg0n/jHQeYA+/PgIlU2dfjdvezCxhQhLZ2Xy5n0X8vANs+h2uPi3Z4tY9Pt/8caOGow59VWh/9hbzxUPv8earVXce8kEXvrqeUMS7EeFh4bw6BfyuXBiCt9bvY2VRfYh27dS/iAgw31lkZ3YiFAun+q9KzzVidlChKtnZ/LmfZ/jN9fl0d7j4O5niljyyPu8vav2MyHf1u3ge6u3cdsTHxEXGcpLXzmP+y6baMkFUxGhNv50yxzOG5fMd1ZuZc0WvROWGj4CrlumvdvBa9urWZKXQVR44IxtD3ShthCWz8li6awMVpdU8vt39nHHU5vJy0rgG5dOZP6kFD78+AjfXrkVe2MnX/7cWO67bKLl1x9Ehtn4v1sLuP0vH/HNFVsJs4WwcEa6pTUpNRQCLtxf215DR48zIMe2B4NQWwjXFWRz9exMVhXb+Z93yvjik5sYnxrL/vo2Ro+I5sUvz6Mgd4TVpR4TFW7jidvP4tYnPuKe50sIDXEPA1UqmAVct0xybDhL8jIoyEmyupRhLcwWwg1njead++fz38tmEBkWwm3zcnnt3gv8KtiPiokI5S9fPItpGfF89a/FvDsEs3IqZSU53YkxXykoKDCbN2+2ZN9q+Gru6OWmxzeyr66NP99WwAUTUqwuSakBEZEiY0zB6dYLuJa7UmciITqMZ+84m7HJMdz19GY27G+wuiSlfELDXQ07STHhPHvn2WQlRXPHU5vYXH7E6pICjtNl2F3TwopNFfz8td2U2pusLkkdR7tl1LBV19LFDY9tpL61m2fumMvs0b49j9PjcFHR2MHH9e2UN7QTGxHKhLQ4JqTFEh8ZdvoNWMQYQ2VTJ1srmtlqb2JLRRPbK5vp8MxWKgLGwMIZo/jmZZMYnzrwexSo/utvt4yGuxrWqps7ueFPG2ns6OH5u85hembCGW3P6TJUNXXy8eH2Tz3KG9qxN3biPMktAdMTIhmfGsvEtDgmpsW6Qz81ljgLQr+po4et9ma2VjS5H/YmDrf1ABBuC2FqRjyzshPJy04gLyuR5LgI/vzPj3n8nwfo7HVy7Zws7r10IpmJevW4L2i4K9VP9sYObvjTRtp7HPz1znOYmhF/yvWNMdS1dnPA0wLvG+KHGjrocbqOrRsTbiM3OYYxfR65yTHkjoyhrcvB3tpW9ta1sq+2jX11rZTVtR27EQu4Q39CWhwTPcE/Pi3Wq6Hf1etkR1UzWyrcYV5qb6K8oQNwt8jHp8SSl53ofmQlMHlU/Elv09jQ1s3/rt/PMxsPgoGbz8nhqxeNO+mtEtXgaLgrNQCHGjq4/k8b6HG6eOHuc5iQGktjR+8nLe/jWuFHuyTAPdVBzojoTwX40UdKXMSAZi11ugz2xg72esJ+X20be2vdod/t+CT0MzyhP8ET+hM8rf3YiJNfuuJ0Gcrq2tha0cQWu7tVvqemFYfn00R6QiR5WZ4gz05gRmbCoP6IVDV18ru39vFiUQVRYTbuuGAsd10wxpJPIcFIw12pATpQ38YNj22ks8eJLURo7uw99potRMhOivpsK3xkDBmJUT6/wUnf0N9b28q+2lb21bV9JvQzE6M83TvusI8Ot7HN3syWiia29eknj4sMdXeteG4Ek5ed2O/bG/bX/vo2HnpjL3/fVk1SdBhfmT+eW+blWH7VcqDTcFdqEMrqWnn4rX0kRIV9qhslOyn6pN0RVnK6DBVHOtyBX9fGvtpW9ta2sb/+k9A/UT95bj9vz+gN2+zN/HLdbv657zCj4iO599IJXDcny9J79wYyDXelhrGjod/W7WBiWpxf/GHasL+BX67bTcmhJsYkx/DNyyayaIZ3bgYznGi4K6X8jjGGt3bV8et1e9hT28q0jHi+dcUk5k9MCcg7qllBr1BVSvkdEeGyqWmsvfcCfntDHi1dvXzxL5u44U8b9WIyL9NwV0oNOVuIcM3sLN7+5nx+unQaHze0c+0fN3DHk5vYWdVidXlBQbtllFKW6+hx8OQH5fxx/X5aux1cNTODb142kVwv30Q9GGifu1Iq4DR39PKn9/bzl/fL6XW6uOGsbO65ZILXh2kGMg13pVTAqmvt4pF3ynj+o0PYQoTbzs3lujnZZCVFDftx8hruSqmAd6ihg9++tZeXtlRyNKpGxoSTmRRFRkIUGYlRZCRGkpV09PsoRsaEB/XIGw13pVTQ2F/vnjahqqmTyqYuz9dOKhs76ex1fmrdiNAQMhKjyPQE/9HQz/J8HZUQGdCt//6Ge8DdQ1UpNfyMS4llXMpnpxI2xtDc2Xss6KuaOqlq7jr2fP2eeupauz/zvuTYCDKToshMjDz2CSAzKYq0+EhiI2xEh4cSHe7+6g8XgA2G18JdRBYAvwNswOPGmJ97a9tKKXUiIkJidDiJ0eFMyzjxdM3dDie1zd3Ymzqo8rT6j7b8d9e08s7uuk/NxHm8MJv0CXsbMRHu72PCQ4nyfI2O6PvcRnREqHv58e+J+OQ9vp6PyCvhLiI24FHgMsAObBKRl40xO72xfaWUGqyIUBujR0YzemT0CV83xtDY0UtlYyd1rV209zjp7HHQ3u2ko8fhee6kvdtBR4+T9h4HHd1Oalu76Oj+5Hl7j4OTTNf/GT9dOo1b5uV67yBPwFst97lAmTHmAICIvAAsBTTclVJ+TUQYERPOiJhwYPA3azHG0O1wuf8AeP4QdPQ4PvW8vcdBZ4+TOTkjvHcAJ+GtcM8EKvo8twNnH7+SiNwN3A0wevRoL+1aKaWsJyJEhtmIDLN5/lBYy1tnCk7UefSZDyjGmMeMMQXGmIKUlBQv7VoppdTxvBXudiC7z/MsoMpL21ZKKTVAXhnnLiKhwF7gEqAS2ATcZIzZcYr31AMHB7nLZODwIN/rb/RY/E+wHAfosfirMzmWHGPMabs+vNLnboxxiMjXgHW4h0I+capg97xn0P0yIrK5P4P4A4Eei/8JluMAPRZ/NRTH4rVx7saYtcBab21PKaXU4AXmpVdKKaVOKVDD/TGrC/AiPRb/EyzHAXos/srnx2LZxGFKKaV8J1Bb7moYE5H1ItIoIhFW16KUv9JwVwFFRHKBC3BfJLdkCPerM6iqgBJw4S4iC0Rkj4iUicgDVtczWCKSLSLvisguEdkhIvdaXdOZEBGbiJSIyKs+3tWtwEbgSeC2PvuPEpHfiMhBEWkWkX+JSJTntfNF5AMRaRKRChG53bN8vYjc2Wcbt4vIBhFZKSK7RcSIyK9FZB+wz7PO7zzbaBGRIhG54Lh/g++JyH4RafW8ni0ij4rIb/oehIi8IiLf8N0/E4jIfZ7fre0i8ryIBMy96kTkCRGpE5HtfZaNEJE3RWSf52uSlTX2x0mO41ee369SEVktIok+2bkxJmAeuMfQ7wfGAuHAVmCq1XUN8ljSgXzP93G4LwILyGPxHMM3gb8Cr/p4P2XAV4A5QC+Q5ln+KLAe9zxHNuBcIAIYDbQCnwfCgJHALM971gN39tn27UDt0WW4Px28C4wAojzLbvZsIxS4H6gBIj2vfRvYBkzCPSVHnmfdubiv2A7xrJcMdByt3Uf/TpnAx33qXgHcbvXvyQDq/xyQD2zvs+yXwAOe7x8AfmF1nYM8jsuBUM/3v/DVcQRay/3Y7JPGmB7g6OyTAccYU22MKfZ83wrswv0fMuCISBawCHjcx/s5H8gBVhhjinD/ob9JREKALwH3GmMqjTFOY8wHxphu4AvAW8aY540xvcaYBmPMlpPsIhL3tIB/7rPsp8aYI8aYTgBjzLOebTiMMb/B/QdkkmfdO4H/MMbsMW5bPet+BDTjvoIb4EZgvTGm1lv/NicRCkR5upSiCaApQYwx7wFHjlu8FHjK8/1TwNVDWtQgnOg4jDFvGGMcnqcbcU/X4nWBFu4nmn0yIAOxL08/8mzgQ2srGbSHge8AJ7/jgXfcBrxhjDl62fZfPcuScQfz/hO8J/sky08kFfengb+ISIln2acuEReR+z1dac0i0oT7j0FyP/b1FO5WP56vz/SzpkExxlQCvwYOAdVAszHmDV/ucwikGWOqwd04wv3zCnRfAl7zxYYDLdz7NftkIBGRWKAQ+IYxpsXqegZKRBYDdZ6WtC/3EwVcD1woIjUiUgPch7vrIx3oAsad4K0VJ1kO0I67RXtUChAL/MEYM9uz7O4+NVwAfNdTR5IxJhF3i/zo7+Wp9vUssFRE8oApwEsnP9oz5+mPXgqMATKAGBG5+dTvUkNJRL4POIDnfLH9QAv3oJp9UkTCcAf7c8aYVVbXM0jnAUtEpBx3N9nFIvKsD/ZzNeAEpgKzPI8pwD9xn2R9AnhIRDI8JzbneYZKPgdcKiLXi0ioiIwUkVmebW4BlolItIiMx9211G2M6fsJalqf7+Nw/2esB0JF5IdAfJ/XHwd+KiITxG2miIwEMMbYcU+o9wxQeLSbx4cuBT42xtQbY3qBVbjPQwSyWhFJB/B8rbO4nkETkduAxcAXjKfz3dsCLdw3ARNEZIyIhOPuu3zZ4poGRUQEd9/uLmPMQ1bXM1jGmAeNMVnGmFzcP493jDG+aCHeBvzFGHPIGFNz9AE8grtf/QHcJzM34e7j/AXuE5iHgIW4T34ewR3oeZ5t/hbowX0S9SngaaBbRCZ9slvK+ny/DvdH6L24ZzTt4tPdhA/hPnH5BtCC++cb1ef1p4AZ+LhLxuMQcI7nD5fg7u/fNQT79aWX+WSE1G3AGgtrGTRx32/6u8ASY0yHz/bjoz8aPiMiC3H38R6dffL/WVzSoHhODv4TdyAd7av+nnFPwBaQRGQ+8C1jzGKraxksT6v+cdyjsQ4AXzTGNHpp25/D3T2Ta4zx9fkJROTHwA24P22U4B4F1O3r/XqDiDwPzMd9PqMW+E/cXVkrcI+AOgRcZ4w5/qSrXznJcTyI+0R8g2e1jcaYf/P6vgMt3JUKRJ4uuBeArcaYn1hdjwp+p+2WOdEg/ONeFxH5vbgvKioVkXzvl6lU4BKRKUAT7hO/D1tcjhom+tPn/iSw4BSvXwlM8DzuBv5w5mUpFTyMMbuMMTHGmHMDcUSUCkynDfeTXEzQ11Lgac9FGxuBxKNntJVSSlnDG5MhnezCoupTvSk5Odnk5uZ6YfdKKTV8FBUVHTZDdA/Vfl9YJCJ347koZPTo0WzevNkLu1dKqeFDRA72Zz1vjHPv94VFxpjHjDEFxpiClJRB3x9bKaXUaXij5f4y8DUReQE4G/ccFqfsklFKqeHC4XTR3NlLU2cvzZ29NHf0Mj41luwR0ad/8xk4bbj3HYQvInbcg/DDAIwxfwTW4r4CsAz3NKZf9FWxSillBWMMbd0Omjo8Ae15HH3e1NlDS9/nfdZr63Z8Zns/vXo6t5yT49OaTxvuxpjPn+Z1A3zVaxUppdQQ6+p1sn5PHRsPHKGxo+dYQLf0aXE7XSe/4DPcFkJCdBiJUWEkRIWRkRjJ5PQ4EqPCSYgKIzHavTzB8zV3ZIzPj0lvHaaUGpZcLsPGjxtYU1LF2u3VtHY5iAnfb3SgAAAVb0lEQVS3kRwX4Q7iqDCykqKOBfPRoD4W4tGfLIsMC8E9hY//0HBXSg0bxhh2VbeyZksla7ZUUdPSRUy4jSumj+Ka2ZnMGzuSUFugzad4YhruSqmgZ2/sYM2WKtZsqWRvbRuhIcL8SSl8f9EULp2SRlS4zeoSvU7DXSkVlJo6evj7tmrWlFTxUbn7IvuCnCR+evV0Fs1IZ0RMuMUV+paGu1IqaHT1OnlrVy0vlVTxj7119DoN41Nj+dblE1k6K9Pnww/9iYa7UiqgOV2GDfsbeGlLJa9vr6Gt20FqXAS3n5vL0lmZTMuI97uTnUNBw10pFXCMMeyoamF1SSWvbK2irrWbuIhQrpw+iqtnZ3LO2JHYQoZfoPel4a6UChiHGjpYs6WSl7ZUsr++nTCbMH9SKtfMzuTiyalEhgXfidHB0nBXSvm1po4eXimt5qWSSooOuu94OHfMCO44fywLZ4wiMTq4T4wOloa7Usrv9Dpd/GNPPYXFdt7eVUeP08XEtFi+s2ASS/IyyEoaPidGB0vDXSnlF472oxcW23l5SxUN7T2MjAnn5nNyWJY/fE+MDpaGu1LKUnUtXby0pZLCokr21LYSbgvhkimpLM/P4sJJKYQFyRWjQ03DXSk15Lp6nbyxs5ZVxXbe21uPy8Cs7ER+evV0rpqZrv3oXqDhrpQaEsYYNh9sZFWxnVdL3RN1ZSRE8u/zx3HN7CzGp8ZaXWJQ0XBXSvlUxZEOVhVXsqrEzsGGDqLCbFw5fRTL52Qxb+xIQob5eHRf0XBXSnlda1cvr22robDYzocfu+d1mTd2JF+/eAJXTh9FTIRGj6/pv7BSyiucLsP7ZYdZVWzn9R01dPW6GJMcw7cun8jVszN1+OIQ03BXSp2RsrpWVhZV8lJJJTUtXcRHhrI8P4vlc7KYnZ2owxctouGu1BAxxrCtsplXtlbxxs5aUmIjuHp2JosDcHRIY3sPL2+torDYTqm9GVuIMH9iCj9YPJVLpug0AP5A3LdAHXoFBQVm8+bNluxbqaFijGFPbSuvbK3i1dJqDjZ0EGYTzhufTFVTJ3tr2wi3hXDR5BSumZ3FRZNTiAj1z2Dscbh4d08dhUV23t3jnk53ano8y/IzWTork5S4CKtLHBZEpMgYU3C69bTlrpQPHKhv45Wt1bxaWsW+ujZsIcK540by1fnjuWLaKBKiwz41s+GaLVWs21FLQlQYi2emsyw/k/zRSZZ3aRz9tFFYZOflrVU0dvSSHBvBbfNyWT4niynp8ZbWp05OW+5KeUnFkQ5eLXUH+o6qFkTgrNwRXJWXwZXTR5Ece/KWrcPp4l9lh1ldUsk6z8nInJHRXD0rk2tmZ5KbHDOERwI1zUevGrWzr66N8NAQLpuaxrX5WVwwITlo7jMaiPrbctdwV+oM1LZ08ffSal4praLkUBPgvtLyqrwMFs1IZ1RC5IC32dbt4PXtNawusfPB/gaMgdmjE1k2O5PFMzNI8tHt4Tp7nLyxs4aVRXbeLzuMy8CcnCSW52exaEY6CdFhPtmvGhgNd6V8pKGtm9e21/DKVve9OY2BqenxXJWXweKZ6V69lVt1cydrtlSxutg978rR+cuXzc7k4impZ9w/73IZNpUfobDYztpt7rsYZSZGsTw/k2vysxgzxJ8Y1OlpuCvlRc2dvazb4Q70D/Y34HQZxqXEsCQvk8V56YxL8e2l88YYdla3sLq4kjVbq6hv7SY+MpRFMzNYnp/JnJyB9c8fbGg/dtVoxZFOYsJtLJyRzrL8LM4eM0KvGvVjGu5KnaH2bgdv7arlla3VvLe3nh6ni9EjorkqL53FMzOYPCrOkhOeDqeL9/c3sLrYzrodtXT2OskeEcU1s07d2m7p6mVtaTWFxXY2lTciAuePT2ZZfiZXTBtFdLiOrwgEXg13EVkA/A6wAY8bY35+3OujgaeARM86Dxhj1p5qmxruyh8dae9h44EG/l5azdu7a+nqdZGeEMnimelclZfBjMwEy0ew9NXW7WDd9hpWl1Ty/v7DGM/sisvy3f3z8ZGh/KvsMIXFlbyxo4Zuh4txKTEsn5PFNbMzSU+IsvoQ1AB5LdxFxAbsBS4D7MAm4PPGmJ191nkMKDHG/EFEpgJrjTG5p9quhruyWq/Txe7qVkoqGik51ETJoUbKGzoASI4NZ9GMdBbnZTBndFJAdFPUNHexZkslq0sq2V3TSmiIkBAVRkN7D4nRYSzJy2B5fhYzs/zrD5QaGG+Oc58LlBljDng2/AKwFNjZZx0DHB3wmgBUDaxcpXyvtqWLkkPuIC8+1EipvZluhwuA1LgI8kcn8fm5o8nPSWJ2dmLADfcblRDJly8cx5cvHMfOqhZWl9ipbelm4Yx0v744SvlGf8I9E6jo89wOnH3cOj8C3hCRrwMxwKUn2pCI3A3cDTB69OiB1qpUv3X1OtlR1expkbtb5VXNXQCE20KYnhnPzefkMHt0IrNHJ5GREBlUrdmpGfFMzZhqdRnKQv0J9xP9xh/fl/N54EljzG9EZB7wjIhMN8a4PvUmYx4DHgN3t8xgClbqeMYY7I2dFB/6pHtlZ3ULvU73r1hWUhRzckdwlyfIp6THaStWBb3+hLsdyO7zPIvPdrvcASwAMMZsEJFIIBmo80aRSvXV3u1gq73pWKt8S0Ujh9t6AIgKs5GXncCdF4xldnYis0Ynkho38AuJlAp0/Qn3TcAEERkDVAI3Ajcdt84h4BLgSRGZAkQC9d4sVA1v1c2dPLfxEG/tqmVvbSsuz+e+sSkxXDgx1dO9ksiktLiA6ytXyhdOG+7GGIeIfA1Yh3uY4xPGmB0i8hNgszHmZeB+4P9E5D7cXTa3G6sG0KtjjDGU2pvZUtHERZNSGT0ysG6WYIzho4+P8NSGctbtqMVlDOeOG8nXp01g9uhEZmUnBtxUuUoNFb2IKQg1tHWzuqSSFzfb2VPbemz52WNGcF1Btt/f5qyzx8lLWyp56oNydte0khAVxo1nZXPzOTlevbRfqUCkV6gOMw6ni3/uO8yKzRW8tauWXqdhVnYi1xdkM3dMEut21PLi5grKGzqIDrexaEY6187JYu6YEX4zSqTiSAdPbyjnb5sqaOlyMCU9ntvPzWFJXiZR4XoCVCnQcB82yg+3s2JzBYXF7jHNI2PCWZafyXUF2UxMi/vUusYYig428uJmO6+WVtHe42T0iGiunZPFsnxr7nFpjOFfZYd56oNy3t5dR4gIC6aN4rZzczkr1/r5zJXyNxruQayjx8HabTWs2FzBRx8fIUTgokmpXFeQzcWTUwkPPf0JxY4e97SyK4vc08qKwLnjRnLtnCwWTEv3eUu5rdtBYZGdpzaUc6C+nZEx4dx09mi+cHbOoKbJVWq40HAPMsYYSiqaWLGpgldLq2nrdjAmOYbrCrJYnp9FWvzgA7HiSAeriitZWVxBxZFOYiNCWTwznesKsrx+N6D99W08s+EgK4vstHU7yMtK4LZzc1k0M13HnivVDxruQaK+tZvVJXZWbLZTVtdGVJiNRTPTueGsbAoGOM3r6bhcho/Kj/DiZjtrt1XT2etkbLJ7kqll+YOfZMrpMqzfU8eTH5Tzz32HCbMJi2dmcNu5uczKTvRa/UoNBxruAczhdLF+Tz0rNlfwzu46HC7DnJwkri/IYtHMDGKHYKRLW7eDtduqWbnZzkfl7q6f8yekcO2cLC6fmtavu9s3d/TyYlEFT284yKEjHaTFR/CFs3P4/NzRejNlpQZJwz0AldW18WJRBauKK6lv7SY5NoLl+ZlcV5DF+NS402/AR8oPt1NYbKewyE5VcxfxkaFclZfBdQXZ5J1ghsHdNS089cFBXiqppLPXyVm5Sdx2bi5XTBtFmF5gpNQZ0XAPEG3dDtaWVrNicwWbDzZiCxEumpTKDWdlM39Sil+Foctl2HCggRc3V/Dadvfc4BNSY7l2ThZLZmWw5VATT20oZ+OBI0SEhnD1rExuPTeHaRkJVpeuVNDQcPcDDqeLIx09HG7t4XBb97FHQ1sP9W3d1Ld2U3SwkY4eJ2NTYrihIJtr8jMDYi6Ulq5e/l5azcoiO0UHG48tz0yM4pZ5OdxQkO2zGzkrNZxpuPtIt8PJ4bYeGo6Gdas7qI+G9ich3kNjRw8n+ucNDw0hJTaC5NhwpqTH+2RUylDaX9/G69trGJ8ay6VT0rAFwI0tlApU3rxZx7Djchle2FTBruoWGtq7j7W869u6ae1ynPA9sRGhJMeGMzI2gjHJMZyVO4JkT4Anx0aQHBdx7HlsRGjABvmJjEuJ5asXjbe6DKVUHxruJ7CqpJLvrd5GQlQYKXHuQJ6aEf/psI6NYGSf7/XyeKWUP9FwP05zZy//vXYXc3KSePHL8wLi3plKKXU8Dffj/OaNPTR29PD00rka7EqpgOU/4+z8wPbKZp7deJBb5+Xq8D2lVEDTcPdwuQz/8dJ2RsREcN9lE60uRymlzoiGu8eLRRVsqWjiewsnkxAVZnU5Sil1RjTcgaaOHn7+2m7m5o7gmtmZVpejlFJnTMMd+OW6PbR0OfjJ1dOCavy5Umr4GvbhvrWiiec/OsTt5+YyeVS81eUopZRXDOtwd7oMP1iznZTYCL5x6QSry1FKKa8Z1uH+wqZDlNqb+f6iKcRF6klUpVTwGLbhfqS9h1++vod5Y0eyJC/D6nKUUsqrhm24/+K13bR3O/jJUj2JqpQKPsMy3IsPNfK3zRXccf4YJqRZd4cjpZTylWEX7k6X4QcvbWdUfCT3XKInUZVSwalf4S4iC0Rkj4iUicgDJ1nnehHZKSI7ROSv3i3Te5778CA7qlr4weKpxAzBjaaVUsoKp003EbEBjwKXAXZgk4i8bIzZ2WedCcCDwHnGmEYRSfVVwWeivrWbX63bw/njk1k4Y5TV5SillM/0p+U+FygzxhwwxvQALwBLj1vnLuBRY0wjgDGmzrtlesfPX9tNV6+TH+tJVKVUkOtPuGcCFX2e2z3L+poITBSR90Vko4gsONGGRORuEdksIpvr6+sHV/EgbSo/QmGxnbsuGMu4lNgh3bdSSg21/oT7iZq4x9/2ORSYAMwHPg88LiKJn3mTMY8ZYwqMMQUpKSkDrXXQHE4XP3hpO5mJUXztYr3Xp1Iq+PUn3O1Adp/nWUDVCdZZY4zpNcZ8DOzBHfZ+4ekNB9ld08oPFk8lOlxPoiqlgl9/wn0TMEFExohIOHAj8PJx67wEXAQgIsm4u2kOeLPQwapr6eKhN/dy4cQUrpiWZnU5Sik1JE4b7sYYB/A1YB2wC1hhjNkhIj8RkSWe1dYBDSKyE3gX+LYxpsFXRQ/Ef63dRY/DxY+X6ElUpdTw0a8+CmPMWmDtcct+2Od7A3zT8/AbGw808NKWKu65eDy5yTFWl6OUUkMmaK9Q7XW6+OGa7WQlRfHv8/UkqlJqeAnas4tPvl/O3to2Hr+1gKhwm9XlKKXUkArKlntNcxcPv7WXSyanculUPYmqlBp+gjLcf/b3nThchh8tmWZ1KUopZYmgC/f3yw7zamk1X5k/nuwR0VaXo5RSlgiqcO9xuE+i5oyM5ssXjrW6HKWUskxQnVD9878+Zn99O3/54llEhulJVKXU8BU0LffKpk5+//Y+Lp+axkWT/HLGYaWUGjJBE+4/e3UnBsMPr5pqdSlKKWW5oAj3f+yt57XtNXz94glkJelJVKWUCvhw73Y4+dHLOxibHMOdF4yxuhyllPILAX9C9f/eO8DHh9t5+ktziQjVk6hKKQUB3nKvONLBI++WsXDGKD43cehu/qGUUv4uoMP9J6/uJESE/1ikJ1GVUqqvgA33d3bX8ubOWu65ZAIZiVFWl6OUUn4lIMO9q9fJj17eybiUGL50np5EVUqp4wXkCdU//mM/h4508Nc7zyY8NCD/PimllE8FXDIebGjnf9fv56q8DM4dn2x1OUop5ZcCLtzXbKkiLET4/sIpVpeilFJ+K+C6Zb5+8XiumZ3JqIRIq0tRSim/FXAtdxHRedqVUuo0Ai7clVJKnZ6Gu1JKBSExxlizY5F64OAg354MHPZiOVbSY/E/wXIcoMfir87kWHKMMaedb8WycD8TIrLZGFNgdR3eoMfif4LlOECPxV8NxbFot4xSSgUhDXellApCgRruj1ldgBfpsfifYDkO0GPxVz4/loDsc1dKKXVqgdpyV0opdQoa7kopFYQCLtxFZIGI7BGRMhF5wOp6BktEskXkXRHZJSI7ROReq2s6EyJiE5ESEXnV6lrOhIgkishKEdnt+dnMs7qmwRKR+zy/W9tF5HkRCZgJmUTkCRGpE5HtfZaNEJE3RWSf52uSlTX2x0mO41ee369SEVktIom+2HdAhbuI2IBHgSuBqcDnRSRQ77HnAO43xkwBzgG+GsDHAnAvsMvqIrzgd8DrxpjJQB4BekwikgncAxQYY6YDNuBGa6sakCeBBcctewB42xgzAXjb89zfPclnj+NNYLoxZiawF3jQFzsOqHAH5gJlxpgDxpge4AVgqcU1DYoxptoYU+z5vhV3iGRaW9XgiEgWsAh43OpazoSIxAOfA/4MYIzpMcY0WVvVGQkFokQkFIgGqiyup9+MMe8BR45bvBR4yvP9U8DVQ1rUIJzoOIwxbxhjHJ6nG4EsX+w70MI9E6jo89xOgAZiXyKSC8wGPrS2kkF7GPgO4LK6kDM0FqgH/uLpYnpcRGKsLmowjDGVwK+BQ0A10GyMecPaqs5YmjGmGtyNIyDV4nq84UvAa77YcKCFu5xgWUCP5RSRWKAQ+IYxpsXqegZKRBYDdcaYIqtr8YJQIB/4gzFmNtBOYHz0/wxPf/RSYAyQAcSIyM3WVqX6EpHv4+6efc4X2w+0cLcD2X2eZxFAHzWPJyJhuIP9OWPMKqvrGaTzgCUiUo67m+xiEXnW2pIGzQ7YjTFHP0GtxB32gehS4GNjTL0xphdYBZxrcU1nqlZE0gE8X+ssrmfQROQ2YDHwBeOji40CLdw3ARNEZIyIhOM+QfSyxTUNiogI7r7dXcaYh6yuZ7CMMQ8aY7KMMbm4fx7vGGMCsoVojKkBKkRkkmfRJcBOC0s6E4eAc0Qk2vO7dgkBenK4j5eB2zzf3wassbCWQRORBcB3gSXGmA5f7Segwt1zEuJrwDrcv6grjDE7rK1q0M4DbsHd0t3ieSy0uijF14HnRKQUmAX8l8X1DIrn08dKoBjYhvv/esBcvi8izwMbgEkiYheRO4CfA5eJyD7gMs9zv3aS43gEiAPe9Py//6NP9q3TDyilVPAJqJa7Ukqp/tFwV0qpIKThrpRSQUjDXSmlgpCGu1JKBSENd6WUCkIa7kopFYT+P1o7tI9gh8LZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time for GD is:\t2.853341579437256\n"
     ]
    }
   ],
   "source": [
    "## 对新的数据集进行 SGD\n",
    "import time\n",
    "input_dim = 784\n",
    "num_classes = 2\n",
    "mini_batch = 8000\n",
    "model4 = LogisticRegression(input_dim, num_classes, lr=1000, batch=1000, weight_scale=4e-2, weight_decay=1.0, reg=0.0)\n",
    "\n",
    "tstart = time.time()\n",
    "model4.train(new_train_data[0:mini_batch], new_train_labels[0:mini_batch])\n",
    "tend = time.time()\n",
    "\n",
    "# 画图\n",
    "Loss, Acc = model4.history()\n",
    "plt.figure\n",
    "plt.subplot(2,1,1)\n",
    "plt.plot(Loss)\n",
    "plt.title('Loss')\n",
    "plt.subplot(2,1,2)\n",
    "plt.plot(Acc)\n",
    "plt.title('Accuracy')\n",
    "plt.show()\n",
    "print('Time for GD is:\\t%s' % (tend - tstart))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 计算相应的参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy:\t0.9668367346938775\n",
      " precision:\t0.9673135852911133\n",
      " recall:\t0.9663265306122449\n",
      " F1-score:\t0.9668198060234813\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>[prediction]</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>[label]</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>947</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>32</td>\n",
       "      <td>948</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "['prediction']    0    1\n",
       "[label]                 \n",
       "0               947   33\n",
       "1                32  948"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 计算 GD 的这些参数\n",
    "import pandas as pd\n",
    "pred_y = model3.test(test_data)\n",
    "\n",
    "tp = 0\n",
    "tn = 0\n",
    "fp = 0\n",
    "fn = 0\n",
    "\n",
    "# 认为 0 是正例，1 是反例\n",
    "for i in range(len(pred_y)):\n",
    "    if pred_y[i] == 0 and pred_y[i] == test_labels[0][i]:\n",
    "        tp += 1\n",
    "    elif pred_y[i] == 1 and pred_y[i] == test_labels[0][i]:\n",
    "        tn += 1\n",
    "    elif pred_y[i] == 0 and pred_y[i] != test_labels[0][i]:\n",
    "        fp += 1\n",
    "    elif pred_y[i] == 1 and pred_y[i] != test_labels[0][i]:\n",
    "        fn += 1\n",
    "\n",
    "# 计算相应参数\n",
    "accuracy = (tp + tn) / (tp + tn + fp + fn)\n",
    "precision = tp / (tp + fp)\n",
    "recall = tp / (tp + fn)\n",
    "F1_score = 2 / (1 / precision + 1 / recall)\n",
    "\n",
    "confusion_matrix = pd.DataFrame(\n",
    "    {\n",
    "        '0': [tp, fp],\n",
    "        '1': [fn, tn]\n",
    "    },\n",
    "    index = ['0','1']\n",
    ")\n",
    "confusion_matrix.columns.name = ['prediction']\n",
    "confusion_matrix.index.name = ['label']\n",
    "print(\n",
    "    'accuracy:\\t%s\\n' % accuracy,\n",
    "    'precision:\\t%s\\n' % precision,\n",
    "    'recall:\\t%s\\n' % recall,\n",
    "    'F1-score:\\t%s\\n' % F1_score\n",
    ")\n",
    "confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy:\t0.9622448979591837\n",
      " precision:\t0.9728601252609603\n",
      " recall:\t0.9510204081632653\n",
      " F1-score:\t0.9618163054695563\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>[prediction]</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>[label]</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>932</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>26</td>\n",
       "      <td>954</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "['prediction']    0    1\n",
       "[label]                 \n",
       "0               932   48\n",
       "1                26  954"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 计算 SGD 的这些参数\n",
    "import pandas as pd\n",
    "pred_y = model4.test(test_data)\n",
    "\n",
    "tp = 0\n",
    "tn = 0\n",
    "fp = 0\n",
    "fn = 0\n",
    "\n",
    "# 认为 0 是正例，1 是反例\n",
    "for i in range(len(pred_y)):\n",
    "    if pred_y[i] == 0 and pred_y[i] == test_labels[0][i]:\n",
    "        tp += 1\n",
    "    elif pred_y[i] == 1 and pred_y[i] == test_labels[0][i]:\n",
    "        tn += 1\n",
    "    elif pred_y[i] == 0 and pred_y[i] != test_labels[0][i]:\n",
    "        fp += 1\n",
    "    elif pred_y[i] == 1 and pred_y[i] != test_labels[0][i]:\n",
    "        fn += 1\n",
    "\n",
    "# 计算相应参数\n",
    "accuracy = (tp + tn) / (tp + tn + fp + fn)\n",
    "precision = tp / (tp + fp)\n",
    "recall = tp / (tp + fn)\n",
    "F1_score = 2 / (1 / precision + 1 / recall)\n",
    "\n",
    "confusion_matrix = pd.DataFrame(\n",
    "    {\n",
    "        '0': [tp, fp],\n",
    "        '1': [fn, tn]\n",
    "    },\n",
    "    index = ['0','1']\n",
    ")\n",
    "confusion_matrix.columns.name = ['prediction']\n",
    "confusion_matrix.index.name = ['label']\n",
    "print(\n",
    "    'accuracy:\\t%s\\n' % accuracy,\n",
    "    'precision:\\t%s\\n' % precision,\n",
    "    'recall:\\t%s\\n' % recall,\n",
    "    'F1-score:\\t%s\\n' % F1_score\n",
    ")\n",
    "confusion_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据平衡前后比较\n",
    "数据平衡前后无论是时间还是分类器在 test 上的表现都差不多. 感觉应该是数据平衡后分类器的表现会好一些，结果真是出人意料。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "torch",
   "language": "python",
   "name": "torch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
